Image area identification method and device, equipment, medium and program product

Abstract

The invention discloses an image area identification method and device, equipment, a medium and a program product, and relates to the field of artificial intelligence. The method comprises the following steps: by taking a first text and an ith first candidate region as denoising conditions, executing a denoising process on noise data corresponding to the ith first candidate region to obtain an ith denoising result; and based on the denoising results corresponding to the plurality of first candidate areas, determining an image area conforming to the text description of the first text from the plurality of first candidate areas. The correlation degree between the image and the text is predicted by adopting a generation method, namely, the image and the text are used as denoising conditions, and correlation information between the image and the text is generated in a mode of gradually removing noise in data, so that semantic relations between visual contents of the image and the first text on different levels are gradually captured; the correlation information between the generated image and the text is more accurate, and the accuracy of the finally determined image area is further improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular, to a method, device, equipment, medium and program product for identifying an image region. Background Art

[0002] In the field of large-scale image processing and computer vision, the visual grounding task of large-scale images (such as high-resolution images) is a very challenging task. The visual grounding task aims to identify the area in a large-scale image that best matches the given text description based on the given text.

[0003] In the related technology, for the visual grounding task of large-scale images, it is necessary to downsample the large-scale image, and then analyze the downsampled image and given text through the visual grounding model, determine the candidate area that meets the given text description in the downsampled image, and finally locate the image area corresponding to the candidate area from the original large-scale image.

[0004] However, the downsampling process will lose image details, and the loss of image details will reduce the model's ability to determine the area in the image that matches the text description, thereby reducing the accuracy of the image area identified by the model that matches the text description. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device, medium and program product for recognizing an image region, which can improve the accuracy of the region recognized from the image that meets the text description. The technical solution is as follows:

[0006] In one aspect, a method for identifying an image region is provided, the method comprising:

[0007] Acquire a plurality of first candidate regions from a first image based on a first text, wherein the plurality of first candidate regions are regions in the first image associated with the first text;

[0008] Obtaining noise data corresponding to each of the plurality of first candidate regions;

[0009] Taking the first text and the i-th first candidate region as denoising conditions, performing a denoising process on the noise data corresponding to the i-th first candidate region to obtain an i-th denoising result, wherein the i-th denoising result is used to characterize the degree of association between the first text and the i-th first candidate region, where i is a positive integer;

[0010] Based on the denoising results respectively corresponding to the plurality of first candidate regions, an image region that meets the text description of the first text is determined from the plurality of first candidate regions.

[0011] In another aspect, a device for identifying an image region is provided, the device comprising:

[0012] An acquisition module, configured to acquire a plurality of first candidate regions from a first image based on a first text, wherein the plurality of first candidate regions are regions in the first image associated with the first text;

[0013] The acquisition module is further used to acquire noise data corresponding to the plurality of first candidate regions respectively;

[0014] a denoising module, configured to perform a denoising process on noise data corresponding to the i-th first candidate region using the first text and the i-th first candidate region as denoising conditions, to obtain an i-th denoising result, wherein the i-th denoising result is used to characterize the degree of association between the first text and the i-th first candidate region, where i is a positive integer;

[0015] A determination module is used to determine an image area that meets the text description of the first text from the multiple first candidate areas based on the denoising results respectively corresponding to the multiple first candidate areas.

[0016] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement any of the above-mentioned image area recognition methods.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement any of the above-mentioned image area recognition methods.

[0018] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the above-mentioned methods for recognizing image regions.

[0019] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0020] On the one hand, a two-stage text-image matching method is adopted. In the first stage, multiple candidate areas associated with the first text are determined from the image to be identified to reduce the search range and reduce the interference of irrelevant areas. In the second stage, a more refined secondary search is performed on the multiple candidate areas, so that the image area that meets the text description of the first text can be more accurately identified. On the other hand, a generation method is used to predict the degree of association between the image and the text, that is, with the image and the text as denoising conditions, the association information between the image and the text is generated by gradually removing the noise in the data, so as to gradually capture the semantic connection between the visual content of the image and the first text at different levels, so that the generated association information between the image and the text is more accurate, and the accuracy of the image area finally determined is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application;

[0023] Figure 2 is a schematic diagram of an image region recognition method provided by an exemplary embodiment of the present application;

[0024] Figure 3 is a flow chart of a method for identifying an image region provided by an exemplary embodiment of the present application;

[0025] Figure 4 is a flow chart of a method for identifying an image region provided by another exemplary embodiment of the present application;

[0026] Figure 5 is a flow chart of a method for identifying an image region provided by another exemplary embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a solution to a visual grounding task for large-scale images provided by an exemplary embodiment of the present application;

[0028] Figure 7 is a structural block diagram of an image region recognition device provided by an exemplary embodiment of the present application;

[0029] Figure 8 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0031] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there any limitation on quantity and execution order.

[0032] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user alone or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the first text, first image, etc. involved in this application are all obtained with full authorization.

[0033] Figure 1 1 is a block diagram of a computer system provided by an exemplary embodiment of the present application. The computer system can be implemented as a system architecture of an image region recognition method. The computer system includes: a terminal 110 and a server 120. The terminal 110 is connected to the server 120 via a wireless network or a wired network.

[0034] The terminal 110 may be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (vehicle computer), a wearable device, a PC (Personal Computer), etc. The terminal 110 may be installed with a client that runs a target application, and the target application may be an application for image area recognition, or an application that provides an image area recognition function, which is not limited in this application. In addition, this application does not limit the form of the target application, including but not limited to App (Application) installed in the terminal 110, applets, etc., and may also be in the form of a web page.

[0035] The server 120 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server that provides basic cloud computing services, or a node in a blockchain system. The server 120 can be a background server of the above-mentioned target application, used to provide background services for the client of the target application.

[0036] In the method for identifying an image region provided in the embodiment of the present application, the execution subject of each step may be a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. Figure 1 As shown, the method for recognizing an image area may be executed by the terminal 110 (for example, a client of a target application installed and running in the terminal 110 executes the method for recognizing an image area), or the method for recognizing an image area may be executed by the server 120, or the method for recognizing an image area may be executed by the terminal 110 and the server 120 interacting and cooperating with each other, and this application does not limit this.

[0037] Optionally, in the method for recognizing an image region provided in an embodiment of the present application, the execution entity of each step may be a chip in a computer device, such as a graphics processing unit (GPU).

[0038] Visual grounding is an important task in computer vision. The purpose of visual grounding is to determine the area in an image that best matches a given text description. Visual grounding is very important in many practical applications, such as image retrieval, image annotation, question-answering systems, and robot navigation.

[0039] Processing large-scale images usually requires powerful computing power, efficient algorithms, and specific data structures. In related technologies, for the visual grounding task of large-scale images, it is necessary to downsample the large-scale image; after obtaining the downsampled image, the downsampled image and the given text are analyzed through the visual grounding model, and the candidate area that matches the given text description is determined in the downsampled image; finally, the image area corresponding to the candidate area is located from the original large-scale image. However, the downsampling process will lose image details, and the loss of image details will affect the model's ability to determine the area matching the text description in the image, thereby reducing the positioning accuracy of the visual grounding task.

[0040] Based on this, the present application embodiment provides a method for identifying an image region. For example, please refer to Figure 2 , Figure 2 A schematic diagram showing a method for identifying an image region is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0041] Step 1: Downsample image a to obtain image b.

[0042] Image a may be a large-scale image. In the embodiments of the present application, a large-scale image generally refers to an image with a higher resolution and / or a larger size. High resolution generally means that the image has a larger number of pixels. For example, an image with a 4K resolution (3840x2160 pixels) or higher resolution can be considered to be large-scale. Large size refers to the size of the image in physical dimensions. For example, an image with a size of 10000x8000 pixels can also be called a large-scale image.

[0043] Downsampling image a means reducing the number of pixels in image a according to certain rules, such as downsampling high-resolution image a by average pooling (merging each 2x2 pixel block of image a into one pixel whose value is the average of the four pixel values) to obtain a low-resolution image b.

[0044] Step 2: Input image b and given text into the visual grounding model 201 to obtain k candidate regions in image a.

[0045] Here, k is an integer greater than 1.

[0046] After inputting the image b and the given text into the visual grounding model 201, the visual grounding model 201 first encodes the image b to obtain a feature map and encodes the given text to obtain a text vector; then the visual grounding model 201 fuses the feature map and the text vector to obtain a multimodal representation, and predicts multiple candidate regions based on the multimodal representation, and assigns a confidence to each candidate region according to the correlation between each candidate region and the given text; finally, the visual grounding model 201 can output the k candidate regions with the highest confidence, and these k candidate regions represent the k regions in the image b that best match the given text description.

[0047] After k candidate regions are determined in image b, for each candidate region, the corresponding position of image a is cropped to obtain k candidate regions in image a.

[0048] Step 3: Determine k denoising conditions based on the k candidate regions in the given text and image a.

[0049] For each candidate region, the image feature representation corresponding to the candidate region is extracted, and the text feature representation corresponding to the text is extracted, and the image feature representation and the text feature representation are fused to obtain the fused feature representation as the denoising condition of the candidate region.

[0050] Step 4: Input k denoising conditions and random noise into the diffusion model 202 to obtain the text-region probability distribution.

[0051] For each candidate region, the fused feature representation and random noise are input into the generated diffusion model. The generated diffusion model will gradually remove the noise data in the random noise under the guidance of the fused feature representation, so as to finally obtain the matching probability between the candidate region and the text.

[0052] After traversing the k candidate regions, the matching probabilities between the k candidate regions and the given text are obtained. These multiple matching probabilities are the text-region joint distribution.

[0053] Step 5: Determine the image region that matches the given text description from image a based on the text-region joint distribution.

[0054] After obtaining the text-region joint distribution, the candidate region with the highest matching probability is determined as the image region that meets the given text description.

[0055] It should be noted that the execution body of the image area recognition method shown in the above steps 2 to 4 can be implemented as a GPU in a computer device. Schematically, the GPU is used to implement the reasoning process of the visual grounding model 201 and the generated diffusion model 202. In some embodiments, the GPU can also be used for the training process of the visual grounding model 201 and the generated diffusion model 202.

[0056] In summary, the image region recognition method provided in the embodiment of the present application first performs a preliminary search based on the downsampled overall image to obtain multiple candidate regions corresponding to the text description, and then crops the multiple candidate regions from the original image; then the generative diffusion model is used to analyze the text description and the multiple candidate regions to generate a text-image joint distribution. On the one hand, the global information of the downsampled image and the local details of the candidate region are combined to more accurately find the region corresponding to the text description in the original image. On the other hand, the generative diffusion model can capture the subtle relationship between the text description and the image region, thereby further improving the positioning accuracy.

[0057] Next, the specific process of the image region recognition method provided by the present application is described.

[0058] Combined with the above introduction, Figure 3 is a flowchart of a method for identifying an image region provided by an embodiment of the present application. The method is executed by a computer device, which may be Figure 1 The terminal 110 and / or the server 120 are shown. The method includes the following steps 310 to 340.

[0059] Step 310 : acquiring a plurality of first candidate regions from a first image based on the first text.

[0060] The first text is a piece of text with specific semantics, such as a product name, a place description, or a person's name. The first text can be a sentence, a phrase, or a keyword, etc., which is not limited here. In some embodiments, the first text is text input or selected by a user in the target application, such as: the user directly inputs or selects the first text in an application interface that provides a recognition function for an image area.

[0061] The first image refers to an image to be processed. For example, in a visual grounding task, it is necessary to identify an area that matches the first text description from the first image. The first image can be a photo, a screenshot, a hand-drawn picture, an image frame in a video, etc., which is not limited here.

[0062] The plurality of first candidate regions are regions in the first image that are associated with the first text. Schematically, the plurality of first candidate regions are regions in the first image that are associated with the semantics of the first text.

[0063] Optionally, the method for acquiring multiple first candidate regions from the first image based on the first text further includes at least one of the following methods:

[0064] Method 1: extracting a text feature representation corresponding to a first text, and extracting an image feature representation corresponding to a first image; fusing the text feature representation and the image feature representation to obtain a fused feature representation; based on the fused feature representation, determining multiple first candidate regions and confidences corresponding to the multiple first candidate regions, respectively, wherein the confidence represents the semantic relevance between the first candidate region and the first text; determining a first candidate region among the multiple first candidate regions that meets the confidence requirements as the first candidate region associated with the first text.

[0065] Method 2: Divide the first image to obtain multiple first candidate areas corresponding to the first image; perform semantic analysis on the multiple first candidate areas to obtain area texts corresponding to the multiple first candidate areas, where the area texts are used to describe the image content of the first candidate areas; based on the similarities between the multiple area texts and the first text, determine the first candidate area among the multiple first candidate areas that meets the similarity requirements as the first candidate area associated with the first text.

[0066] It should be noted that the above method description of obtaining multiple first candidate areas from the first image based on the first text can refer to the method description of obtaining multiple second candidate areas from the second image based on the first text in step 512 below, which will not be repeated here.

[0067] In some embodiments, the resolution of the first image is greater than a preset resolution, where the preset resolution is a pre-set larger resolution, that is, the first image is a large-scale image. Processing large-scale images usually requires powerful computing power, efficient algorithms and specific data structures. In this embodiment, for large-scale images, it is necessary to downsample the large-scale image, match the text of the downsampled image, and then determine the area associated with the first text in the original large-scale image.

[0068] Optionally, a downsampling process is performed on the first image to obtain a second image; multiple second candidate regions are determined from the second image based on the first text, and the multiple second candidate regions are regions in the second image associated with the first text; multiple first candidate regions are determined from the first image based on the multiple second candidate regions, and the multiple first candidate regions correspond one-to-one to the multiple second candidate regions, wherein there is a mapping relationship between the first position of the i-th first candidate region in the first image and the second position of the i-th second candidate region in the second image.

[0069] Schematically, in the first image and the second image, each pixel has a unique coordinate position. Due to the downsampling process, there is a mapping relationship between the pixel coordinates in the second image and the corresponding pixel coordinates in the first image, and this mapping relationship can map the second candidate area in the second image back to the first image, thereby obtaining the first candidate area.

[0070] Step 320: Obtain noise data corresponding to each of the plurality of first candidate regions.

[0071] Optionally, the noise data refers to random noise. Schematically, this random noise refers to the noise probability value between the first candidate area and the first text. After removing the noise in the noise probability value, the matching probability between the first candidate area and the first text can be obtained, that is, the matching score described below, which is used to characterize the degree of association between the first candidate area and the first text.

[0072] In some embodiments, a group of random variables are randomly sampled from a preset noise distribution as noise data corresponding to the plurality of first candidate regions respectively.

[0073] Optionally, the noise distribution can be implemented as at least one of Gaussian distribution, Laplace distribution, uniform distribution, etc., which is not limited here. Schematically, multiple noise values ​​are randomly extracted from the Gaussian distribution, and the multiple values ​​are assigned to each first candidate region, and each first candidate region has one or more noise values ​​as noise data.

[0074] It should be noted that the matching probability between the first candidate area and the first text is a value between 1 and 1 (including 0 and 1), that is, the noise data also needs to be limited to between 0 and 1 (including 0 and 1), then the noise value randomly drawn from the Gaussian distribution is between 0 and 1 (including 0 and 1).

[0075] Alternatively, the noise data is pre-set data, and schematically, one or more noises are pre-assigned to each first pre-selected area.

[0076] Step 330 , taking the first text and the i-th first candidate region as denoising conditions, performing a denoising process on the noise data corresponding to the i-th first candidate region to obtain an i-th denoising result.

[0077] The i-th denoising result is used to characterize the correlation degree between the first text and the i-th first candidate region, where i is a positive integer.

[0078] In some embodiments, the i-th denoising result includes a matching score between the first text and the i-th first candidate region, and the matching score is used to express the degree of association between the first text and the i-th first candidate region. Schematically, the matching score can be implemented as a probability value, such as 0.8, etc. The higher the matching score, the higher the degree of association between the first text and the i-th first candidate region. Quantifying the degree of association between the first text and the i-th first candidate region through the matching score helps to quickly query the image area that meets the text description of the first text.

[0079] In some embodiments, the first text, the i-th first candidate region and the noise data corresponding to the i-th first candidate region are input into a generated diffusion model, and the i-th denoising result is output; wherein the generated diffusion model is a machine learning model that learns the correlation relationship between the input text and the image area during the training phase.

[0080] Diffusion Models are machine learning models used to generate data. They start from a noise sample (i.e., the noise data mentioned above) and gradually remove the noise part of the sample through multiple iterations to obtain a denoising result. In each iteration, the diffusion model removes the noise part of the current noise sample to obtain a denoising result, which is the noise sample in the next iteration.

[0081] In the embodiment of the present application, the generated diffusion model is a trained model, and its main task is to learn the association relationship between the input text and the image area, and generate the association information between the input text and the image area, such as: matching probability, matching score, etc., and the denoising process of the generated diffusion model is carried out under the prompt of the denoising condition, that is, in each iteration, the generated diffusion model will obtain the multimodal information composed of the first text and the first candidate area as conditional information, and guide the generated diffusion model to denoise the noise data, thereby generating the association information between the input text and the image area.

[0082] The generative diffusion model is used to gradually remove noise from the data to generate association information between the image and the text. During the denoising process, the generative diffusion model can gradually capture the semantic connection between the visual content of the image and the first text at different levels, making the generated association information more accurate and further improving the accuracy of the final determined image area.

[0083] In some embodiments, text and image features are extracted, and the noise data is denoised using the text and image features. Optionally, the above process of obtaining the i-th denoising result further includes the following steps 1 and 2.

[0084] Step 1: extract a first text feature representation corresponding to the first text, and extract an i-th image feature representation corresponding to the i-th first candidate region.

[0085] Illustratively, a first text feature representation corresponding to a first text is extracted through a text encoder, and the text encoder can be implemented as BERT, LSTM, etc.; an i-th image feature representation corresponding to the i-th first candidate region is extracted through an image encoder, and the image encoder can be implemented as a convolutional neural network, a visual Transformer, etc.

[0086] Step 2: Perform a denoising process on the noise data corresponding to the i-th first candidate region based on the first text feature representation and the i-th image feature representation to obtain the i-th denoising result.

[0087] Optionally, the first text feature representation and the i-th image feature representation are fused to obtain an i-th fused feature representation; and a denoising process is performed on noise data corresponding to the i-th first candidate region based on the i-th fused feature representation to obtain an i-th denoising result.

[0088] The method of fusing text and image features includes at least one of the following methods:

[0089] Method 1: Concatenate the first text feature representation and the i-th image feature representation to obtain the i-th fused feature representation.

[0090] In schematic form, assuming that the first text feature representation is a vector of length T, and the i-th image feature representation is a vector of length I, then the fused feature representation after splicing is a vector of length T + I. The fusion of text and image features by feature splicing is simple and efficient, which improves the efficiency of feature fusion.

[0091] Method 2: The first text feature representation and the i-th image feature representation are fused through the attention mechanism to obtain the i-th fused feature representation.

[0092] Optionally, an attention mechanism is used to determine an attention weight between the first text feature representation and the i-th image feature representation; the first text feature representation and the i-th image feature representation are fused according to the attention weight to obtain an i-th fused feature representation.

[0093] Schematically, the similarity between the first text feature representation and the i-th image feature representation is calculated through the attention mechanism. Common similarity metrics include dot product, cosine similarity, etc. The similarity is normalized by the softmax function to obtain the attention weight. The i-th image feature representation is weighted according to the attention weight to obtain the weighted image feature representation. The i-th image feature representation is concatenated or added with the first text feature representation to obtain the i-th fused feature representation. The fusion of text and image features using the attention mechanism can give different importance to different image regions, so that the subtle relationship between the image and text can be better captured, which is conducive to improving the recognition accuracy of image regions.

[0094] After obtaining the i-th fused feature representation, a denoising process is performed on the noise data corresponding to the i-th first candidate region by generating a diffusion model based on the i-th fused feature representation to obtain an i-th denoising result.

[0095] In the above embodiment, by extracting the text feature representation corresponding to the first text and the image feature representation corresponding to the i-th first candidate region as denoising conditions, cross-modal matching is achieved through the denoising process, thereby improving the accuracy of cross-modal matching.

[0096] Among them, by extracting and fusing text features and image features, effective integration of cross-modal information is achieved. This fusion not only retains the rich information of text and image, but also captures the potential correlation between the two through joint representation, providing more comprehensive input for the subsequent denoising process and further improving the accuracy of cross-modal matching.

[0097] In some embodiments, the generated diffusion model includes a feature extraction layer and a denoising layer, wherein the feature extraction layer is used to extract and fuse a first text feature representation corresponding to the first text and an i-th image feature representation corresponding to the i-th first candidate region to obtain an i-th fused feature representation; the denoising layer is used to perform a denoising process on the noise data corresponding to the i-th first candidate region with the i-th fused feature representation as the denoising condition to obtain an i-th denoising result.

[0098] In some embodiments, the generated diffusion model is a pre-trained diffusion model. The following takes the denoising result as a matching score as an example to introduce the training process of the generated diffusion model. Optionally, the training process of the generated diffusion model includes the following steps:

[0099] Step 1: Obtain multiple candidate regions from the sample image based on the sample text.

[0100] The multiple candidate regions are regions in the sample image that are associated with the sample text. The process of obtaining multiple candidate regions from the sample image based on the sample text can be referred to step 310, which will not be described in detail here.

[0101] Step 2: Based on the reference region associated with the sample text in the sample image, obtain the distances between multiple candidate regions and the reference region as reference matching scores.

[0102] In schematic form, the coordinate distance between the center point coordinates of each candidate region and the center point coordinates of the reference region is calculated as the distance between the candidate region and the reference region. Alternatively, the candidate region is divided into multiple sub-regions, and the distance between the candidate region and the reference region is determined based on the distances between the multiple sub-regions and the reference region, assuming that the candidate region is divided into sub-region 1 and sub-region 2, the distance between the center point coordinates of sub-region 1 and the center point coordinates of the reference region is calculated to be 2, the distance between the center point coordinates of sub-region 2 and the center point coordinates of the reference region is calculated to be 4, the average of the two distances is calculated to be 3 (or other statistical values, such as maximum value, mode, etc.), and the distance 3 is used as the distance between the candidate region and the reference region.

[0103] After determining the distance between the candidate region and the reference region, a reference matching score is determined according to the distance, wherein the distance and the reference matching score are positively correlated. Multiple candidate regions are traversed to obtain reference matching scores corresponding to the multiple candidate regions.

[0104] Step 3: Define multiple time steps. In the multiple time steps, gradually add noise to the reference matching scores corresponding to the multiple candidate regions, and gradually convert the reference matching scores corresponding to the multiple candidate regions into pure noise.

[0105] Schematically, at each time step, Gaussian noise is added to the reference matching score, and the noise level increases with the increase of time steps until the reference matching score completely becomes pure noise.

[0106] In each time step, the reference matching score after adding noise is the sample noise corresponding to the time step. For example, in the tth time step, noise t is added to the reference matching score, and the reference matching score after adding noise t is the sample noise corresponding to the tth time step.

[0107] Step 4: In the tth time step, taking the first text and multiple candidate regions as denoising conditions, a denoising process is performed on multiple sample noises corresponding to the tth time step through a sample diffusion model to obtain multiple sample matching scores.

[0108] Where t is a positive integer. For the i-th candidate region, the first text and the i-th candidate region are used as denoising conditions. The i-th sample noise corresponding to the t-th time step is denoised through the sample diffusion model to obtain the i-th sample matching score, where the i-th sample noise is the sample noise corresponding to the i-th candidate region.

[0109] Schematically, the sample diffusion model includes a feature extraction layer and a denoising layer, wherein the feature extraction layer is used to extract and fuse the sample text feature representation corresponding to the first text and the i-th sample image feature representation corresponding to the i-th candidate region to obtain the i-th fused feature representation; the denoising layer is used to gradually denoise the sample noise corresponding to the i-th candidate region using the i-th fused feature representation as the denoising condition to obtain the i-th sample matching score.

[0110] By traversing multiple candidate regions and performing the above denoising process, multiple sample matching scores can be obtained.

[0111] Step 5: Determine the target loss based on the differences between multiple sample matching scores and multiple reference matching scores.

[0112] Schematically, multiple sample matching scores can be regarded as predicted probability distributions of text-multiple candidate regions, and multiple reference matching scores can be regarded as reference probability distributions of text-multiple candidate regions. Based on the difference between the predicted probability distribution and the reference probability distribution, the target loss is determined.

[0113] Optionally, the loss function for calculating the target loss may implement at least one of a cross entropy loss function, a Kullback-Leibler divergence (KL divergence) loss function, a contrastive learning loss function, etc., which are not limited here. The loss function is implemented as a KL divergence loss function and a contrastive learning loss function as an example for explanation.

[0114] The KL divergence loss function is used to calculate the distribution loss between the reference probability distribution and the predicted probability distribution, so that the predicted probability distribution generated by the denoising layer is as close to the reference probability distribution as possible.

[0115] The contrastive learning loss function is used to shorten the distance between similar text-candidate regions and to extend the distance between dissimilar text-candidate regions, where similar text-candidate regions can be defined as text-candidate regions with predicted matching scores greater than a preset score, and dissimilar text-regions can be defined as text-candidate regions with predicted matching scores less than or equal to a preset score. The contrastive learning loss is determined based on the similar text-candidate region distances and dissimilar text-candidate region distances among multiple candidate regions.

[0116] In some embodiments, after obtaining the distribution loss and the contrastive learning loss, the distribution loss and the contrastive learning loss are fused to obtain the target loss.

[0117] Step 6: Train the sample diffusion model according to the target loss to obtain the generated diffusion model.

[0118] Optionally, if the target loss is obtained by fusing the distribution loss and the contrastive learning loss, the sample diffusion model is updated through the back propagation algorithm with the goal of minimizing the target loss; or, the denoising layer of the sample diffusion model is updated through the back propagation algorithm with the goal of minimizing the distribution loss, and the feature extraction layer of the sample diffusion model is updated through the back propagation algorithm with the goal of minimizing the contrastive learning loss.

[0119] When the target loss obtained by the sample diffusion model is less than the preset loss or the training times of the generated diffusion model reaches the preset times, the training is stopped. The sample diffusion model obtained at this time is the generated diffusion model.

[0120] Step 340 : determining an image region that matches the text description of the first text from among the plurality of first candidate regions based on the denoising results respectively corresponding to the plurality of first candidate regions.

[0121] Illustratively, multiple first candidate regions are traversed, and the above step 330 is executed to obtain denoising results corresponding to the multiple first candidate regions respectively.

[0122] In some embodiments, the i-th denoising result includes a matching score between the first text and the i-th first candidate region.

[0123] Optionally, a first candidate region with the highest matching score among the multiple first candidate regions is taken as the image region; or a first candidate region with a matching score greater than a preset score among the multiple first candidate regions is taken as the image region.

[0124] Among them, the matching score can be realized as a value between 0 and 1 (including 0 and 1). Schematically, assuming that there is candidate area 1 with a matching score of 0.5; candidate area 2 with a matching score of 0.7; candidate area 3 with a matching score of 0.9, then the candidate area 3 with the highest matching score is selected as the image area that conforms to the text description of the first text; alternatively, the preset score is set to 0.8, and the candidate area 3 with a matching score greater than 0.8 is obtained as the image area that conforms to the text description of the first text.

[0125] In the above embodiment, by introducing a matching score to evaluate the correlation between the first text and each first candidate region, it is possible to more accurately determine which image region best matches the text description.

[0126] In some embodiments, if it is determined that there are multiple regions that match the text description of the first text, the image region is determined based on the positional relationship between the multiple regions. For example, a minimum bounding rectangle (or a minimum bounding box of any other shape) is drawn around the multiple regions, and the bounding box is used as the image region.

[0127] In some embodiments, after the image region is determined, the image region and the first text may be input into a regression model, and a region in the image region that more accurately matches the text description of the first text may be determined through the regression model.

[0128] Schematically, the image region and the first text are input into a regression model, wherein the regression model may be a deep neural network, such as a combination of a convolutional neural network and a recurrent neural network. Optionally, the regression model has been trained on a large amount of annotated data to learn the correspondence between the image region and the text description. In the regression model, a convolutional neural network is used to extract visual features from the image region; a recurrent neural network is used to extract semantic features from the first text; the extracted visual features and semantic features are fused to obtain fused features, and the fused features are analyzed by the regression model to output a more precise region location, which is usually a sub-region in the image region and has a higher degree of match with the text description of the first text, that is, higher than the degree of match between the original image region and the text description of the first text.

[0129] In summary, the method for identifying image regions provided in the embodiment of the present application, on the one hand, adopts a two-stage text-image matching method. In the first stage, multiple candidate regions associated with the first text are determined from the image to be identified to reduce the search range and reduce the interference of irrelevant regions. In the second stage, a more refined secondary search is performed on the multiple candidate regions, so that the image region that conforms to the text description of the first text can be more accurately identified. On the other hand, a generation method is used to predict the degree of association between the image and the text, that is, with the image and the text as denoising conditions, the association information between the image and the text is generated by gradually removing the noise in the data, thereby gradually capturing the semantic connection between the visual content of the image and the first text at different levels, so that the generated association information between the image and the text is more accurate, further improving the accuracy of the image region finally determined.

[0130] In some embodiments, the first candidate region may be divided into a plurality of sub-regions, so that the matching score of the first candidate region is determined according to the matching scores of the sub-regions. Figure 4 , the above Figure 3 The illustrated embodiment may also be implemented as steps 410 to 440 as follows.

[0131] Step 410 : acquiring a plurality of first candidate regions from a first image based on the first text.

[0132] The plurality of first candidate regions are regions in the first image that are associated with the first text. Schematically, the plurality of first candidate regions are regions in the first image that are associated with the semantics of the first text.

[0133] Step 421 , dividing a plurality of first candidate regions into a plurality of sub-regions corresponding to the plurality of first candidate regions respectively.

[0134] Optionally, the plurality of first candidate regions are divided respectively according to a preset division strategy to obtain a plurality of sub-regions respectively corresponding to the plurality of first candidate regions.

[0135] Illustratively, the preset division strategy is associated with the shapes of the multiple first candidate regions. Taking the multiple first candidate regions as rectangles as an example, each first candidate region can be evenly divided into a plurality of squares.

[0136] Step 422: Obtain noise data corresponding to each of the plurality of first candidate regions.

[0137] Optionally, the noise data refers to random noise. Schematically, the random noise refers to noise probability values ​​between the first text and the plurality of sub-regions corresponding to the first candidate region.

[0138] In some embodiments, a group of random variables are randomly sampled from a preset noise distribution as noise data corresponding to the plurality of first candidate regions respectively.

[0139] Optionally, the above noise distribution can be implemented as at least one of Gaussian distribution, Laplace distribution, uniform distribution, etc., which is not limited here. Schematically, for the i-th first candidate region, a plurality of noise values ​​are randomly extracted from the Gaussian distribution, and the plurality of values ​​are distributed one by one to a plurality of sub-regions corresponding to the i-th first candidate region, each sub-region has a noise value, and finally the noise values ​​corresponding to the plurality of sub-regions are obtained as the noise data corresponding to the i-th first candidate region.

[0140] Step 431, taking the first text and the i-th first candidate region as denoising conditions, performing a denoising process on the noise data corresponding to the i-th first candidate region, and obtaining matching scores between the first text and a plurality of sub-regions corresponding to the i-th first candidate region.

[0141] Schematically, the matching score is used to express the degree of association between the first text and the sub-region. Schematically, the matching score can be implemented as a probability value, such as 0.8, etc. The higher the matching score, the higher the degree of association between the first text and the sub-region.

[0142] Optionally, taking the first text and the i-th first candidate region as denoising conditions, a denoising process is performed on the noise data corresponding to the i-th first candidate region by generating a diffusion model to obtain matching scores between multiple sub-regions corresponding to the i-th first candidate region and the first text.

[0143] In some embodiments, a first text feature representation corresponding to the first text is extracted, and an i-th image feature representation corresponding to the i-th first candidate region is extracted; the first text feature representation and the i-th image feature representation are fused to obtain an i-th fused feature representation; a denoising process is performed on the noise data corresponding to the i-th first candidate region based on the i-th fused feature representation to obtain matching scores between multiple sub-regions corresponding to the i-th first candidate region and the first text.

[0144] Step 432 : determining a matching score between the ith first candidate region and the first text based on matching scores between the first text and a plurality of sub-regions corresponding to the ith first candidate region.

[0145] Optionally, the matching scores between the multiple sub-regions and the first text are fused to obtain the matching score between the i-th first candidate region and the first text.

[0146] Illustratively, the average of the matching scores between the multiple sub-regions and the first text is calculated (or other statistical values, such as the maximum value, the mode, etc., which are not limited here), and the average is used as the matching score between the i-th first candidate region and the first text.

[0147] In some embodiments, regional weights of the multiple sub-regions are determined based on the matching scores between the multiple sub-regions and the first text respectively, and the matching scores between the multiple sub-regions and the first text are fused based on the regional weights to obtain the matching score between the i-th first candidate region and the first text.

[0148] Schematically, the regional weight of each sub-region is obtained, and the regional weight is positively correlated with the matching score corresponding to the sub-region; based on the regional weight, the weighted average of the matching scores between multiple sub-regions and the first text is calculated, and the weighted average is used as the matching score between the i-th first candidate region and the first text.

[0149] Step 440 : determining an image region that matches the text description of the first text from among the plurality of first candidate regions based on the matching scores corresponding to the plurality of first candidate regions respectively.

[0150] Illustratively, multiple first candidate regions are traversed to obtain matching scores corresponding to the multiple first candidate regions respectively, and the first candidate region with the highest matching score among the multiple first candidate regions is used as the image region.

[0151] In summary, the image region recognition method provided in the embodiment of the present application can divide the first candidate region into multiple sub-regions, thereby determining the matching score of the first candidate region based on the matching scores of the sub-regions, so as to determine the image region that meets the text description of the first text from the multiple first candidate regions. By subdividing the candidate region into smaller sub-regions, the degree of matching between each sub-region and the first text description can be more accurately evaluated, which helps to capture more detailed features and differences, thereby reducing the possibility of misjudgment and improving the accuracy of the overall matching.

[0152] In some embodiments, the first image may be downsampled, and a plurality of first candidate regions may be obtained from the first image according to the downsampled image. Figure 5 , the above Figure 3 The illustrated embodiment may also be implemented as the following steps 511 to 540 .

[0153] Step 511, performing a downsampling process on the first image to obtain a second image.

[0154] Optionally, a resolution of the first image is greater than a preset resolution.

[0155] Downsampling refers to converting an image from a high resolution to a low resolution. The downsampling process usually involves reducing the number of pixels in the image, that is, the number of pixels in the first image is greater than the number of pixels in the second image. Optionally, the downsampling method for the first image includes at least one of the following methods:

[0156] Method 1: Direct Averaging Method. Divide the first image into multiple non-overlapping small regions according to a predetermined size (such as 2x2, 3x3, etc.); for each small region, calculate the average value of all pixel values within the region, and this average value will be used as the pixel value at the corresponding position in the downsampled image (i.e., the center or the specified position such as the upper left corner of the small region, depending on the implementation). By combining multiple pixels into one pixel value (i.e., the region average value), the size (width and height) of the image will be reduced according to the size ratio of the divided regions. For example, if each region is 2x2 pixels, the width and height of the downsampled image will be half of the original image.

[0157] Method 2: Maximum Value Method. Divide the first image into multiple non-overlapping small regions according to a predetermined size (such as 2x2, 3x3, etc.); for each small region, calculate the maximum value of all pixel values within the region. This maximum value will be used as the pixel value at the corresponding position in the downsampled image (i.e., the center or the specified position such as the upper left corner of the small region, depending on the implementation).

[0158] Method 3: Interpolation Method. Taking bilinear interpolation as an example, for the interpolation point p in the first image, calculate its corresponding coordinates in the first image, and find the 4 pixels closest to this coordinate on the first image. These 4 pixels usually form a rectangle and are located at the four corners of the rectangle respectively. Suppose the coordinates of these four points are Q11(x1, y1), Q12(x1, y2), Q21(x2, y1), and Q22(x2, y2) respectively, where x1 < x < x2 and y1 < y < y2. In the X direction, perform linear interpolation on Q11 and Q21 to obtain the pixel value of point R1. Similarly, perform linear interpolation on Q12 and Q22 to obtain the pixel value of point R2; in the Y direction, use the pixel values of R1 and R2 obtained in the previous step to perform linear interpolation to obtain the pixel value of the interpolation point p in the first image.

[0159] It should be noted that the above examples of the downsampling methods for the first image are only illustrative and are not limited here.

[0160] Step 512, determine multiple second candidate regions from the second image based on the first text.

[0161] The multiple second candidate regions are the regions in the second image associated with the first text. Schematically, the multiple second candidate regions are the regions in the first image semantically associated with the first text.

[0162] Optionally, the method for obtaining multiple second candidate regions from the second image based on the first text further includes at least one of the following methods:

[0163] Method 1: extract the second text feature representation corresponding to the first text, and extract the target image feature representation corresponding to the second image; fuse the second text feature representation and the target image feature representation to obtain a fused feature representation; based on the fused feature representation, determine multiple second candidate regions and the confidences corresponding to the multiple second candidate regions from the second image, the confidences are used to characterize the semantic relevance between the second candidate regions and the first text; determine the second candidate regions that meet the confidence requirements among the multiple second candidate regions as the second candidate regions associated with the first text.

[0164] Illustratively, a visual grounding model can be used to predict multiple second candidate regions in the second image. The visual grounding model includes a text encoder, an image encoder, a feature fusion layer, and an output layer. The following introduces the process of predicting multiple second candidate regions using the visual grounding model.

[0165] The first text is encoded into a text vector representation through a text encoder, and the text encoder can be implemented as BERT (Bidirectional Encoder Representations from Transformers), LSTM (Long Short-Term Memory), etc. The second image is encoded into a feature map through an image encoder, each position of the feature map corresponds to an area in the image and contains regional image features of the area. The image encoder can be implemented as a convolutional neural network, a visual Transformer, etc.

[0166] After obtaining the text vector representation and feature map, the feature fusion layer concatenates the text vector representation and the feature map to obtain a fused feature map; alternatively, the feature fusion layer fuses the text feature representation and the image feature representation through an attention mechanism to obtain a fused feature map. Each position of the fused feature map contains the cross-modal information of the corresponding image region. The cross-modal information of the image region is the fusion information of the regional image features of the image region and the text vector representation.

[0167] The fused feature map is predicted through the output layer, and a set of bounding box parameters is predicted at each position of the feature map, where the set of bounding box parameters includes the center coordinates, width, height and confidence of the bounding box, wherein the confidence represents the semantic correlation between the image area corresponding to the bounding box and the first text, and the areas corresponding to the k groups of bounding box parameters with the highest confidence among the multiple groups of bounding box parameters are respectively determined as the multiple second candidate areas that meet the confidence requirements, or the areas corresponding to the k groups of bounding box parameters with confidence greater than a preset confidence among the multiple groups of bounding box parameters are respectively determined as the multiple second candidate areas that meet the confidence requirements, and k is an integer greater than 1.

[0168] The above method 1 directly fuses text and image features and uses confidence to screen out the most relevant candidate regions, which can more accurately evaluate the semantic relevance between each candidate region and the text.

[0169] Method 2: Divide the second image to obtain multiple second candidate areas corresponding to the second image; perform semantic analysis on the multiple second candidate areas to obtain regional texts corresponding to the multiple second candidate areas, where the regional texts are used to describe the image content of the second candidate areas; based on the similarities between the multiple regional texts and the first text, determine the second candidate areas among the multiple second candidate areas that meet the similarity requirements as the second candidate areas associated with the first text.

[0170] Illustratively, the second image is divided into multiple smaller areas, i.e., multiple second candidate areas, through a preset segmentation strategy, wherein parameters such as area size and number defined in the preset segmentation strategy may be user-defined and are not limited here. For example: the second image is evenly divided into 16 areas; or, 16 coordinate points are obtained in the second image through uniform sampling, and these 16 coordinate points are used as center points, and three rectangular areas are cut out at each coordinate point, and the sizes of the three rectangular areas are different.

[0171] After obtaining multiple second candidate regions, the multiple second candidate regions can be input into the image description generation model. The image description generation model can convert the content of each second candidate region into region text. These region texts are descriptive texts used to explain the content of the region. For example, if the region contains a "cat", then the region text may be "the cat is on the sofa" and so on.

[0172] After determining the regional texts corresponding to the plurality of second candidate regions, the similarity between each regional text and the first text is calculated, for example, the regional text and the first text are converted into text vector representations, and then the vector distance (such as cosine similarity, etc.) is calculated to represent the similarity between the regional text and the first text. After obtaining the similarities corresponding to the plurality of second candidate regions, the k second candidate regions with the highest similarity are used as the plurality of second candidate regions that meet the similarity requirements, or the k second candidate regions with similarity greater than a preset similarity are used as the plurality of second candidate regions that meet the similarity requirements.

[0173] The above method 2 divides the image into multiple regions, analyzes the content of each region and compares the similarity with the text. It can capture local features in the image and help identify details and small targets.

[0174] It should be noted that the above examples of the method for obtaining multiple second candidate regions are only illustrative and are not limited here.

[0175] Step 513: determine a plurality of first candidate regions from the first image based on the plurality of second candidate regions.

[0176] The multiple first candidate regions correspond one-to-one to the multiple second candidate regions, wherein there is a mapping relationship between the first position of the i-th first candidate region in the first image and the second position of the i-th second candidate region in the second image.

[0177] Illustratively, the second candidate region determined in the downsampled image (ie, the second image) is mapped back to the original image (ie, the first image) for further processing or analysis on a higher resolution image.

[0178] Since the second image is a downsampled version of the first image, each pixel or area in the second image corresponds to one or more pixels or a larger area in the first image. This mapping relationship can be recorded during the downsampling process. After text matching is performed on the second image, the second candidate area in the second image can be accurately mapped back to the first image based on the mapping relationship, thereby obtaining the first candidate area corresponding to the second candidate area.

[0179] Step 520: Obtain noise data corresponding to each of the plurality of first candidate regions.

[0180] In some embodiments, a group of random variables are randomly sampled from a preset noise distribution as noise data corresponding to the plurality of first candidate regions respectively.

[0181] Step 530 , taking the first text and the i-th first candidate region as denoising conditions, performing a denoising process on the noise data corresponding to the i-th first candidate region to obtain an i-th denoising result.

[0182] The i-th denoising result is used to characterize the correlation degree between the first text and the i-th first candidate region, where i is a positive integer.

[0183] In some embodiments, the i-th denoising result includes a matching score between the first text and the i-th first candidate region.

[0184] In some embodiments, the information entropy difference between the first image and the second image may be used to assist the denoising process.

[0185] Optionally, an information entropy difference between the first image and the second image is obtained, and the information entropy difference is used to characterize the amount of information loss of the first image compared to the second image after a downsampling process; a first text feature representation corresponding to the first text is extracted, and an i-th image feature representation corresponding to the i-th first candidate region is extracted; a denoising process is performed on the noise data corresponding to the i-th first candidate region based on the first text feature representation, the i-th image feature representation and the information entropy difference to obtain an i-th denoising result.

[0186] Schematically, a method for obtaining the information entropy difference between a first image and a second image is introduced. First, the first image and the second image are read and converted into grayscale images. For each image, the histogram of its grayscale value is calculated. The histogram is a chart representing the number of pixels of each grayscale level in the image. The frequency of each grayscale level in the histogram is divided by the total number of pixels in the image to obtain the probability of each grayscale level. For each grayscale level, its probability is calculated multiplied by the negative of the logarithm of the probability, and the results of all grayscale levels are summed to obtain the information entropy. After obtaining the information entropy corresponding to the first image and the second image respectively, the difference between them is calculated to obtain the information entropy difference.

[0187] After obtaining the information entropy difference, it can be used as an additional feature or weight factor to guide the denoising process. The information entropy difference is used as a weight factor to adjust the importance of image feature representation in the denoising process.

[0188] The information weight corresponding to the i-th image feature representation is determined according to the information entropy difference, and the i-th image feature representation is weighted based on the information weight to obtain the i-th image feature representation after weighted processing; a denoising process is performed on the noise data corresponding to the i-th first candidate region based on the first text feature representation and the i-th image feature representation after weighted processing to obtain the i-th denoising result.

[0189] Among them, the information weight is negatively correlated with the information entropy difference. When the information entropy difference is larger, the information weight is smaller, which means that the difference between the second image and the first image is larger, that is, the second image has lost more details, then the confidence of the first candidate region determined according to the second image is lower, so the decision weight of the first candidate region in the denoising process is reduced. When the information entropy difference is smaller, the information weight is larger, the difference between the second image and the first image is smaller, that is, the second image has lost fewer details, then the confidence of the first candidate region determined according to the second image is higher, so the decision weight of the first candidate region in the denoising process can be appropriately increased.

[0190] In the above embodiment, the information entropy difference can quantify the degree of information loss in the downsampling process, and provide additional reference information for the denoising process. By using the information entropy difference as a weight factor, the importance of different image feature representations in the denoising process can be more accurately adjusted, thereby improving the precision and accuracy of denoising.

[0191] Step 540 : determining an image region that meets the text description of the first text from among the plurality of first candidate regions based on the denoising results respectively corresponding to the plurality of first candidate regions.

[0192] Optionally, a first candidate region with the highest matching score among the multiple first candidate regions is taken as the image region; or a first candidate region with a matching score greater than a preset score among the multiple first candidate regions is taken as the image region.

[0193] To summarize, the image area recognition method provided in the embodiment of the present application performs a downsampling process on the first image to obtain a second image with a lower resolution, and performs the first stage text-image matching process on the second image with a lower resolution. Since the number of pixels to be processed is reduced, the computational complexity is reduced, which helps to improve the processing efficiency of the entire recognition process.

[0194] Schematically, the image region recognition method provided by the embodiment of the present application is described by taking the first image implemented as a large-scale image as an example. Figure 6 , which shows a schematic diagram of a solution for the visual grounding task of large-scale images. The solution can be implemented as follows:

[0195] Step 1: Input the large-scale image into the downsampling module 601 to obtain a thumbnail.

[0196] Schematically, the large-scale image is downsampled through the OpenCV library to obtain a thumbnail with a resolution of 640*640. In the OpenCV library, the main method for downsampling large-scale images is to use the cv2.resize() function. The cv2.resize() function can adjust the size of the image according to the specified size or scaling ratio. For downsampling (i.e. reducing the image size), the target size (width and height) can be directly specified, and the OpenCV library will adjust the image size accordingly to output a thumbnail.

[0197] Step 2: Input the query text and thumbnail into the visual ground model 602 to obtain the coordinate information of k candidate regions.

[0198] Schematically, the query text and thumbnail are input into the visual grounding model, the image feature representation of the thumbnail is obtained using the image encoder in the visual grounding model, the text feature representation of the query text is obtained using the text encoder in the visual grounding model, and the image feature representation and the text feature representation are fused to obtain a fused feature representation.

[0199] The output layer in the visual grounding model is used to predict the fused feature representation to obtain multiple groups of bounding box parameters, each group of bounding box parameters includes the center coordinates, width, height and confidence of the bounding box, wherein the confidence represents the semantic correlation between the image area corresponding to the bounding box and the first text, and the areas corresponding to the k groups of bounding box parameters with the highest confidence among the multiple groups of bounding box parameters are determined as candidate areas that meet the confidence requirements, and the center coordinates, width and height of the bounding boxes corresponding to the k candidate areas are output as the coordinate information of the k candidate areas.

[0200] Step 3: input the coordinate information of the k candidate regions and the large-scale image into the candidate region generation module 603 to obtain k candidate region images.

[0201] Illustratively, after downsampling the large-scale image to obtain a thumbnail, the mapping relationship between the large-scale image and the thumbnail is stored in the candidate region generation module 603, wherein the mapping relationship indicates the positional correspondence between each pixel in the thumbnail and one or more pixels or larger regions in the large-scale image.

[0202] The coordinate information of k candidate regions is input into the candidate region generation module to obtain the coordinates of each pixel point corresponding to the k candidate regions, and the mapping relationship stored in the candidate region generation module is read. According to the mapping relationship, the k candidate region images can be determined in the large-scale image.

[0203] Step 4, input the query text, k candidate region images and k groups of noise data into the generation diffusion model 604 to obtain the text-candidate region image joint distribution.

[0204] Schematically, for a specified candidate area image among k candidate area images, a diffusion model is generated to extract a first text feature representation corresponding to the query text, and a specified image feature representation corresponding to the specified candidate area image; the first text feature representation and the specified image feature representation are fused to obtain a fused feature representation; the diffusion model is generated to perform a denoising process on noise data corresponding to the specified candidate area image based on the fused feature representation to obtain a matching score between the specified candidate area image and the query text.

[0205] Traverse the k candidate region images and obtain the matching scores corresponding to the k candidate region images. The probability distribution of the matching scores corresponding to the k candidate region images and the query text is the joint distribution of text-candidate region images.

[0206] Step 5: input the joint distribution of text and candidate region image into the sorting module 605 to obtain the prediction result.

[0207] Illustratively, the joint distribution of text and candidate region images is input into the sorting module 605, and the sorting module 605 outputs the candidate region with the highest matching score among the multiple candidate regions as the prediction result.

[0208] In summary, the embodiment of the present application provides a solution for the visual grounding task of large-scale images. When processing the visual grounding task of large-scale images, the image is first downsampled, and the candidate area is found using global information. Then, the generative diffusion model is used to finely match the high-resolution candidate area. By introducing the generative diffusion model, the joint probability distribution between the text description and the candidate area is modeled, so that the model can capture richer local detail information and improve the positioning accuracy.

[0209] The image area recognition method provided in the embodiment of the present application can be applied to various scenarios such as XR (Extended Reality), autonomous driving and assisted driving, medical image analysis, security monitoring, etc., and is described below using XR (Extended Reality) applications and medical image analysis as examples.

[0210] Scenario 1: XR application.

[0211] In the embodiment of the present application, XR application refers to an application that combines multiple technical forms such as virtual reality (VR), augmented reality (AR) and mixed reality (MR). These technologies combine digital content with the real world in different ways to provide users with an immersive experience. In XR applications, users can use text input to describe the virtual objects or objects in the real world that they want to find or interact with. The image area recognition method provided in the embodiment of the present application can help XR applications quickly locate the image area in the image. Schematically, the image area recognition method in the XR application includes the following steps:

[0212] Step 1: Get the descriptive text entered by the user in the XR application.

[0213] Optionally, descriptive text can be used to query elements in XR applications, which can be implemented as virtual objects or objects in the real world. For example, in an XR tour guide application, tourists can find and understand the attraction in a virtual or augmented reality view by entering the name or feature description of the attraction. Another example is that in an XR game, if a player enters "find a stone tablet with a star pattern engraved on it", the location of the stone tablet that meets the characteristics can be marked in the player's field of view, and the player's attention can be guided through slight vibration feedback.

[0214] Step 2: Obtain the environment image in the XR application in real time.

[0215] The environment image may be at least one of a real-world image or a virtual environment image in the user's field of view captured in real time by an XR device.

[0216] Step 3: downsample the environment image to obtain a downsampled image.

[0217] Step 4: Input the descriptive text and the downsampled image into the visual grounding model to obtain the coordinate information of k candidate regions.

[0218] Schematically, the query text and thumbnail are input into the visual grounding model, and the areas corresponding to the k groups of bounding box parameters with the highest confidence are output as candidate areas that meet the confidence requirements, and the center coordinates, width, and height of the bounding boxes corresponding to these k candidate areas are output as the coordinate information of the k candidate areas.

[0219] Step 5: cut out k candidate region images from the environment image according to the coordinate information of the k candidate regions.

[0220] In an illustrative manner, a pre-stored mapping relationship is read, and according to the mapping relationship, the coordinates of each pixel point corresponding to the k candidate regions are mapped to the environment image, and k candidate region images are determined in the environment image.

[0221] Step 6: Input the descriptive text, k candidate region images and k groups of noise data into the generated diffusion model to obtain the joint distribution of text-candidate region images.

[0222] Schematically, for a specified candidate region image among the k candidate region images, a diffusion model is generated to perform a denoising process to generate a matching score between the specified candidate region image and the descriptive text; the k candidate region images are traversed to obtain the matching scores corresponding to the k candidate region images respectively, and the probability distribution of the matching scores corresponding to the k candidate region images and the descriptive text is the joint distribution of text-candidate region image.

[0223] Step 7, input the joint distribution of text and candidate region images into the sorting module to determine the image regions that match the text description of the descriptive text.

[0224] Illustratively, the ranking module outputs a candidate region with the highest matching score among the multiple candidate regions as the image region that conforms to the text description of the descriptive text.

[0225] Step 8: Label the image area in the XR application.

[0226] Illustratively, in XR applications, visual feedback (such as highlighting, border marking, or 3D arrow indication) is used to mark the image area in the user's field of view.

[0227] Scenario 2: Medical image analysis.

[0228] In the field of medical imaging, doctors need to find specific lesions or anatomical structures in images based on the patient's specific symptoms. The image region recognition method provided in the embodiment of the present application can assist doctors in quickly locating the area in the image that meets the text description, thereby improving the accuracy and efficiency of diagnosis. Schematically, the image region recognition method used in medical image analysis includes the following steps:

[0229] Step 1: Obtain search text input in the medical imaging system.

[0230] Illustratively, the search text includes at least one of a description of the patient's symptoms, the name of the lesion or anatomical structure to be searched, and the like.

[0231] Step 2: Load the medical image.

[0232] Illustratively, a medical image to be analyzed is loaded, and the medical image may be a high-definition diagnostic image of a corresponding patient, such as a CT image, magnetic resonance imaging, three-dimensional color ultrasound, and the like.

[0233] Step 3: downsample the medical image to obtain a downsampled image.

[0234] Step 4: Input the search text and the downsampled image into the visual grounding model to obtain the coordinate information of k candidate regions.

[0235] Schematically, the search text and thumbnail are input into the visual grounding model, and the areas corresponding to the k groups of bounding box parameters with the highest confidence are output as candidate areas that meet the confidence requirements, and the center coordinates, width, and height of the bounding boxes corresponding to these k candidate areas are output as the coordinate information of the k candidate areas.

[0236] Step 5: cut out k candidate region images from the medical image according to the coordinate information of the k candidate regions.

[0237] In an illustrative manner, a pre-stored mapping relationship is read, and according to the mapping relationship, the coordinates of each pixel point corresponding to the k candidate regions are mapped to the medical image, and k candidate region images are determined in the medical image.

[0238] Step 6: Input the search text, k candidate region images and k groups of noise data into the generated diffusion model to obtain the joint distribution of text-candidate region images.

[0239] Schematically, for a specified candidate region image among the k candidate region images, a diffusion model is generated to perform a denoising process to generate a matching score between the specified candidate region image and the search text; the k candidate region images are traversed to obtain the matching scores corresponding to the k candidate region images respectively, and the probability distribution of the matching scores corresponding to the k candidate region images and the search text is the joint distribution of the text-candidate region image.

[0240] Step 7: Input the joint distribution of text and candidate region images into the sorting module to determine the image regions that match the text description of the search text.

[0241] Illustratively, the ranking module outputs a candidate region with the highest matching score among the multiple candidate regions as the image region that meets the text description of the search text.

[0242] Step 8: label the image area in the medical image.

[0243] Schematically, image areas are directly marked in medical images to provide doctors with intuitive visual assistance, facilitating rapid positioning and diagnosis.

[0244] For illustration, please refer to Figure 7 , which shows a structural block diagram of an image region recognition device, such as Figure 7 As shown, the device comprises:

[0245] An acquisition module 710 is configured to acquire a plurality of first candidate regions from a first image based on a first text, wherein the plurality of first candidate regions are regions in the first image associated with the first text;

[0246] The acquisition module 710 is further used to acquire noise data corresponding to the plurality of first candidate regions respectively;

[0247] A denoising module 720 is configured to perform a denoising process on the noise data corresponding to the i-th first candidate region using the first text and the i-th first candidate region as denoising conditions to obtain an i-th denoising result, wherein the i-th denoising result is used to characterize the degree of association between the first text and the i-th first candidate region, where i is a positive integer;

[0248] The determination module 730 is configured to determine, from among the plurality of first candidate regions, an image region that matches the text description of the first text based on denoising results respectively corresponding to the plurality of first candidate regions.

[0249] In some embodiments, the denoising module 720 is used to input the first text, the i-th first candidate area and the noise data corresponding to the i-th first candidate area into a generated diffusion model, and output the i-th denoising result; wherein the generated diffusion model is a machine learning model that learns the association relationship between the input text and the image area during the training phase.

[0250] In some embodiments, the denoising module 720 is used to extract the first text feature representation corresponding to the first text, and to extract the i-th image feature representation corresponding to the i-th first candidate region; based on the first text feature representation and the i-th image feature representation, a denoising process is performed on the noise data corresponding to the i-th first candidate region to obtain the i-th denoising result.

[0251] In some embodiments, the denoising module 720 is used to fuse the first text feature representation and the i-th image feature representation to obtain the i-th fused feature representation; and perform a denoising process on the noise data corresponding to the i-th first candidate region based on the i-th fused feature representation to obtain the i-th denoising result.

[0252] In some embodiments, the denoising module 720 is used to concatenate the first text feature representation and the i-th image feature representation to obtain the i-th fused feature representation; or, to fuse the first text feature representation and the i-th image feature representation through an attention mechanism to obtain the i-th fused feature representation.

[0253] In some embodiments, the i-th denoising result includes a matching score between the first text and the i-th first candidate region, and the matching score is used to express the degree of association between the first text and the i-th first candidate region; the determination module 730 is used to select the first candidate region with the highest matching score among the multiple first candidate regions as the image region.

[0254] In some embodiments, the i-th first candidate region includes multiple sub-regions; the denoising module 720 is used to perform a denoising process on the noise data corresponding to the multiple sub-regions respectively, with the first text and the i-th first candidate region as denoising conditions, to obtain the matching scores between the multiple sub-regions and the first text respectively; based on the matching scores between the multiple sub-regions and the first text respectively, determine the matching score between the i-th first candidate region and the first text.

[0255] In some embodiments, the acquisition module 710 is used to perform a downsampling process on the first image to obtain a second image; determine multiple second candidate areas from the second image based on the first text, and the multiple second candidate areas are areas in the second image associated with the first text; determine the multiple first candidate areas from the first image based on the multiple second candidate areas, and the multiple first candidate areas correspond one-to-one to the multiple second candidate areas, wherein there is a mapping relationship between the first position of the i-th first candidate area in the first image and the second position of the i-th second candidate area in the second image.

[0256] In some embodiments, the acquisition module 710 is used to extract a second text feature representation corresponding to the first text, and to extract a target image feature representation corresponding to the second image; to fuse the second text feature representation and the target image feature representation to obtain a fused feature representation; based on the fused feature representation, to determine multiple second candidate regions and confidences corresponding to the multiple second candidate regions from the second image, respectively, wherein the confidences are used to characterize the semantic relevance between the second candidate regions and the first text; and to determine a second candidate region among the multiple second candidate regions that meets the confidence requirements as the second candidate region associated with the first text.

[0257] In some embodiments, the acquisition module 710 is used to divide the second image to obtain multiple second candidate areas corresponding to the second image; perform semantic analysis on the multiple second candidate areas to obtain area texts corresponding to the multiple second candidate areas, respectively, and the area texts are used to describe the image content of the second candidate areas; based on the similarities between the multiple area texts and the first text, determine the second candidate areas among the multiple second candidate areas that meet the similarity requirements as the second candidate areas associated with the first text.

[0258] In some embodiments, the acquisition module 710 is used to obtain the information entropy difference between the first image and the second image, and the information entropy difference is used to characterize the amount of information loss of the first image compared to the second image after the downsampling process; the determination module 730 is used to extract the first text feature representation corresponding to the first text and the i-th image feature representation corresponding to the i-th first candidate region; based on the first text feature representation, the i-th image feature representation and the information entropy difference, a denoising process is performed on the noise data corresponding to the i-th first candidate region to obtain the i-th denoising result.

[0259] In summary, the image region recognition device provided by the embodiment of the present application, on the one hand, adopts a two-stage text-image matching method. In the first stage, multiple candidate regions associated with the first text are determined from the image to be recognized to reduce the search range and reduce the interference of irrelevant regions. In the second stage, a more refined secondary search is performed on the multiple candidate regions, so that the image region that conforms to the text description of the first text can be more accurately identified. On the other hand, a generation method is used to predict the degree of association between the image and the text, that is, with the image and the text as denoising conditions, the association information between the image and the text is generated by gradually removing the noise in the data, thereby gradually capturing the semantic connection between the visual content of the image and the first text at different levels, so that the generated association information between the image and the text is more accurate, and the accuracy of the image region finally determined is further improved.

[0260] It should be noted that the specific definition in the embodiment of the recognition device for one or more image regions provided above can refer to the definition of the recognition method for image regions above, which will not be repeated here. Each module of the above device can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of the processor of the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operation of each module.

[0261] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein a computer program is stored in the memory; and the processor is used to execute the computer program in the memory to implement the image area recognition method provided by the above-mentioned method embodiments.

[0262] For example, Figure 8 800 is a structural block diagram of a computer device 800 provided by an exemplary embodiment of the present application. Optionally, the computer device 800 is a server 800.

[0263] Typically, the server 800 includes a processor 801 and a memory 802 .

[0264] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 801 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0265] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, which is used to be executed by the processor 801 to implement the image area recognition method provided by each method embodiment in the present application.

[0266] In some embodiments, the server 800 may also optionally include: an input interface 803 and an output interface 804. The processor 801, the memory 802 and the input interface 803, the output interface 804 may be connected via a bus or a signal line. Each peripheral device may be connected to the input interface 803, the output interface 804 via a bus, a signal line or a circuit board. The input interface 803 and the output interface 804 may be used to connect at least one peripheral device related to input / output (I / O) to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802 and the input interface 803, the output interface 804 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802 and the input interface 803, the output interface 804 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0267] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the computer device 800, and the computer device 800 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0268] In an exemplary embodiment, the present application provides a chip, which includes a programmable logic circuit and / or program instructions. When the chip runs on a computer device, it is used to implement the image area recognition method provided by the above-mentioned various method embodiments.

[0269] In an exemplary embodiment, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is loaded and executed by a processor to implement the image region recognition method provided by each of the above method embodiments.

[0270] In an exemplary embodiment, the present application provides a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the processor of the computer device loads and executes to implement the image region recognition method provided in the above method embodiment.

[0271] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0272] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned computer-readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0273] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0274] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying an image region, characterized in that: The method comprises: Acquire a plurality of first candidate regions from a first image based on a first text, wherein the plurality of first candidate regions are regions in the first image associated with the first text; Obtaining noise data corresponding to each of the plurality of first candidate regions; Taking the first text and the i-th first candidate region as denoising conditions, performing a denoising process on the noise data corresponding to the i-th first candidate region to obtain an i-th denoising result, wherein the i-th denoising result is used to characterize the degree of association between the first text and the i-th first candidate region, where i is a positive integer; Based on the denoising results respectively corresponding to the plurality of first candidate regions, an image region that meets the text description of the first text is determined from the plurality of first candidate regions.

2. The method according to claim 1, characterized in that The step of performing a denoising process on the noise data corresponding to the i-th first candidate region using the first text and the i-th first candidate region as denoising conditions to obtain an i-th denoising result includes: Inputting the first text, the i-th first candidate region and the noise data corresponding to the i-th first candidate region into a diffusion model, and outputting the i-th denoising result; The generative diffusion model is a machine learning model that learns the association relationship between input text and image regions during the training phase.

3. The method according to claim 1, characterized in that The step of performing a denoising process on the noise data corresponding to the i-th first candidate region using the first text and the i-th first candidate region as denoising conditions to obtain an i-th denoising result includes: Extracting a first text feature representation corresponding to the first text, and extracting an i-th image feature representation corresponding to the i-th first candidate region; A denoising process is performed on the noise data corresponding to the i-th first candidate region based on the first text feature representation and the i-th image feature representation to obtain the i-th denoising result.

4. The method according to claim 3, characterized in that The performing a denoising process on the noise data corresponding to the i-th first candidate region based on the first text feature representation and the i-th image feature representation to obtain the i-th denoising result includes: fusing the first text feature representation and the i-th image feature representation to obtain an i-th fused feature representation; A denoising process is performed on the noise data corresponding to the i-th first candidate region based on the i-th fused feature representation to obtain the i-th denoising result.

5. The method according to claim 4, characterized in that The fusing the first text feature representation and the i-th image feature representation to obtain the i-th fused feature representation includes: concatenating the first text feature representation and the i-th image feature representation to obtain the i-th fused feature representation; or, The first text feature representation and the i-th image feature representation are fused through an attention mechanism to obtain the i-th fused feature representation.

6. The method according to any one of claims 1 to 5, characterized in that: The i-th denoising result includes a matching score between the first text and the i-th first candidate region, where the matching score is used to express the degree of association between the first text and the i-th first candidate region; The determining, based on the denoising results respectively corresponding to the plurality of first candidate regions, an image region that meets the text description of the first text from the plurality of first candidate regions comprises: The first candidate region with the highest matching score among the multiple first candidate regions is used as the image region.

7. The method according to claim 6, characterized in that The i-th first candidate area includes multiple sub-areas; The step of performing a denoising process on the noise data corresponding to the i-th first candidate region using the first text and the i-th first candidate region as denoising conditions to obtain an i-th denoising result includes: Taking the first text and the i-th first candidate region as denoising conditions, performing a denoising process on the noise data corresponding to the multiple sub-regions respectively, and obtaining matching scores between the multiple sub-regions and the first text respectively; Based on the matching scores between the multiple sub-regions and the first text respectively, a matching score between the i-th first candidate region and the first text is determined.

8. The method according to any one of claims 1 to 5, characterized in that: The acquiring a plurality of first candidate regions from the first image based on the first text includes: Performing a downsampling process on the first image to obtain a second image; Determine a plurality of second candidate regions from the second image based on the first text, wherein the plurality of second candidate regions are regions in the second image associated with the first text; The multiple first candidate regions are determined from the first image based on the multiple second candidate regions, and the multiple first candidate regions correspond one-to-one to the multiple second candidate regions, wherein there is a mapping relationship between the first position of the i-th first candidate region in the first image and the second position of the i-th second candidate region in the second image.

9. The method according to claim 8, characterized in that The determining a plurality of second candidate regions from the second image based on the first text comprises: Extracting a second text feature representation corresponding to the first text, and extracting a target image feature representation corresponding to the second image; fusing the second text feature representation and the target image feature representation to obtain a fused feature representation; Based on the fused feature representation, determining a plurality of second candidate regions and confidences respectively corresponding to the plurality of second candidate regions from the second image, wherein the confidences are used to characterize the semantic relevance between the second candidate regions and the first text; A second candidate region among the multiple second candidate regions that meets the confidence requirement is determined as the second candidate region associated with the first text.

10. The method according to claim 8, characterized in that The determining a plurality of second candidate regions from the second image based on the first text comprises: dividing the second image to obtain a plurality of second candidate regions corresponding to the second image; Performing semantic analysis on the multiple second candidate regions to obtain region texts corresponding to the multiple second candidate regions respectively, wherein the region texts are used to describe image contents of the second candidate regions; Based on the similarities between the multiple region texts and the first text respectively, a second candidate region that meets the similarity requirement among the multiple second candidate regions is determined as a second candidate region associated with the first text.

11. The method according to claim 8, characterized in that The method further comprises: Acquire an information entropy difference between the first image and the second image, wherein the information entropy difference is used to characterize an amount of information loss of the first image compared to the second image after the downsampling process; The step of performing a denoising process on the noise data corresponding to the i-th first candidate region using the first text and the i-th first candidate region as denoising conditions to obtain an i-th denoising result includes: Extracting a first text feature representation corresponding to the first text, and extracting an i-th image feature representation corresponding to the i-th first candidate region; A denoising process is performed on the noise data corresponding to the i-th first candidate region based on the first text feature representation, the i-th image feature representation and the information entropy difference to obtain the i-th denoising result.

12. A device for identifying an image region, characterized in that: The device comprises: An acquisition module, configured to acquire a plurality of first candidate regions from a first image based on a first text, wherein the plurality of first candidate regions are regions in the first image associated with the first text; The acquisition module is further used to acquire noise data corresponding to the plurality of first candidate regions respectively; a denoising module, configured to perform a denoising process on noise data corresponding to the i-th first candidate region using the first text and the i-th first candidate region as denoising conditions, to obtain an i-th denoising result, wherein the i-th denoising result is used to characterize the degree of association between the first text and the i-th first candidate region, where i is a positive integer; A determination module is used to determine an image area that meets the text description of the first text from the multiple first candidate areas based on the denoising results respectively corresponding to the multiple first candidate areas.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the image area recognition method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the image area recognition method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for recognizing an image area as claimed in any one of claims 1 to 11.

Images