Image area content description method and device, equipment and medium

Abstract

The invention relates to the technical field of visual languages, in particular to an image area content description method and device, equipment and a medium. The method comprises the following steps: acquiring a to-be-described image and an image region specified by description of the to-be-described image, and generating a region mask matrix by using an attention module of a visual encoder in a trained visual language model and combining the image region; when feature coding is carried out on an image to be described, attention weighting is carried out on features of an image region based on the region mask matrix to obtain an image feature vector; and a text encoder in a trained visual language model is used to describe the image feature vector to obtain a content description text, and the trained visual language model is obtained by loss training constructed based on comparison of the difference between the total image description and the regional cut image description. The feature in the region is endowed with a higher weight through the region attention mask to obtain a focused image feature vector, so that the focusing capability of the model is optimized, the global interference is inhibited, and the description accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of visual language technology, and in particular to a method, apparatus, device and medium for describing the content of an image region. Background Technology

[0002] Existing visual-language models (VLMs) typically tend to generate global descriptions based on the entire image when generating image descriptions. However, when faced with a specific image region, they are prone to producing descriptions that are not focused on the user-specified region, and the output may include objects or information unrelated to the target region. This leads to a decrease in the model's performance in tasks requiring fine-grained region description, especially in applications that require precise descriptions of specific objects or regions (e.g., driver assistance, medical image annotation, image question answering, etc.).

[0003] Existing methods lack effective mechanisms to guide the model to focus on a specified region and suppress globally irrelevant features, making it difficult for the model to consistently generate accurate region descriptions during training and inference.

[0004] Therefore, optimizing the focusing ability of visual language models to suppress global interference and improve the accurate description of the execution area has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method, apparatus, device, and medium for describing the content of an image region, in order to solve the problem of how to optimize the focusing ability of a visual language model to suppress global interference and improve the accurate description of the execution region.

[0006] In a first aspect, embodiments of this application provide a method for describing the content of an image region, including: Obtain the image to be described and the image region specified by the description of the image to be described; Using the attention module of the visual encoder in the trained visual language model, combined with the image region, a region mask matrix is ​​generated; When performing feature encoding on the image to be described, attention weighting is applied to the features of the image region based on the region mask matrix to obtain the image feature vector; The image feature vector is described using the text encoder in the trained visual language model to obtain the content description text. The trained visual language model is trained using a loss constructed based on the difference between the full image description and the region cropped image description.

[0007] Secondly, embodiments of this application provide an image region content description device, comprising: The data acquisition module is used to acquire the image to be described and the image region specified by the description of the image to be described; The region masking module is used to generate a region mask matrix by combining the image regions with the attention module of the visual encoder in the trained visual language model. The feature encoding module is used to perform attention weighting on the features of the image region based on the region mask matrix when performing feature encoding on the image to be described, so as to obtain an image feature vector; The content description module is used to describe the image feature vector using a text encoder in a trained visual language model to obtain content description text. The trained visual language model is trained using a loss constructed based on the difference between full-image description and region cropping image description. Thirdly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image region content description method as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image region content description method as described in the first aspect.

[0009] The beneficial effects of the embodiments in this application compared with the prior art are: The process involves acquiring the image to be described and the image region specified for the description of the image to be described. Using the attention module of the visual encoder in a trained visual language model, a region mask matrix is ​​generated by combining the image region. When encoding features of the image to be described, attention weights are applied to the features of the image region based on the region mask matrix to obtain an image feature vector. The text encoder in the trained visual language model is then used to describe the image feature vector to obtain the content description text. The trained visual language model is trained using a loss constructed based on comparing the differences between the full-image description and the region-cropped image description.

[0010] Specifically, by assigning better weights to features within a region through a region attention mask, a focused image feature vector is obtained. The model is trained based on comparing the differences between the full-image description and the region-cropped image description, which optimizes the focusing ability of the visual language model, suppresses global interference, and achieves accurate description of the target region. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for an image region content description method provided in Embodiment 1 of this application; Figure 2 This is a flowchart illustrating an image region content description method provided in Embodiment 2 of this application; Figure 3 This is a flowchart illustrating an image region content description method provided in Embodiment 3 of this application; Figure 4 This is a flowchart illustrating an image region content description method provided in Embodiment 4 of this application; Figure 5 This is a schematic diagram of the structure of an image region content description device provided in Embodiment 5 of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in Embodiment Six of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0015] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0017] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0019] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0020] To illustrate the technical solution of this application, specific embodiments are described below.

[0021] The image region content description method provided in Embodiment 1 of this application can be applied to, for example, Figure 1In this application environment, the client and server communicate. A model is configured on either the client or server; devices without a model can provide data to other devices. Clients include, but are not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0022] See Figure 2 This is a flowchart illustrating an image region content description method provided in Embodiment 2 of this application. The above-described image region content description method can be applied to... Figure 1 The server in the system has a corresponding visual language model deployed on it, and the client can provide image data to the server to train the model.

[0023] like Figure 2 As shown, the image region content description method may include the following steps: Step S201: Obtain the image to be described and the image region specified by the description of the image to be described.

[0024] The image to be described includes, but is not limited to, images captured by image acquisition devices (e.g., cameras, mobile phone cameras, etc.), or images generated by a certain generative model. The image to be described may include one or more objects, colors, lines, etc.

[0025] The image region specified in the description of an image can be understood as a region in the image. The purpose of this region is for the user to understand the image content within the region. For example, if the upper left corner of the image contains the object "chair", then the image content of this region can be described as "there is a chair".

[0026] Here, the specified image area is defined by the user according to their needs. Specifically, it can be a limitation of a region in the image. For example, given the image coordinates, it can be a region enclosed by the coordinates of the four corner points or two opposite corner points. Alternatively, the image can be directly divided into several equal parts, with one part serving as the image area.

[0027] Step S202: Using the attention module of the visual encoder in the trained visual language model, combined with the image regions, a region mask matrix is ​​generated.

[0028] The visual language model includes image encoding, text encoding, cross-modal fusion, and reasoning and generation. It extracts features from images using models such as Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs), which are typically low-dimensional representations of the images. Then, pre-trained language models, such as BERT or GPT, are used to convert the input text into high-dimensional vector representations, capturing the grammatical, semantic, and contextual information of the language. The image and text representations are then fused, with common methods including the use of two-stream neural networks and attention mechanisms. In this way, the model can learn the relationships between the images and text. Based on the fused representations, the model performs reasoning (e.g., image question answering) or generation (e.g., image description). Different tasks use different output strategies and output heads.

[0029] In this embodiment, the visual encoder of the visual language model adopts an attention model formed by an attention mechanism, which can generate a region mask matrix for the corresponding image region by combining the image region, so as to use the region mask matrix for weighting in subsequent image encoding.

[0030] Specifically, the mask matrix of this region can be set so that the element value in the corresponding matrix outside the region is 0, and the element value in the corresponding matrix inside the region is not 0. Of course, the size of the non-zero element value can be calculated based on the features of the region, or it can be a preset value. The specific accuracy and computational consumption requirements are designed accordingly.

[0031] Optionally, the attention module of the visual encoder in the trained visual language model is used to generate a region mask matrix by combining image regions, including: Using the attention module of the visual encoder in the trained visual language model, combined with the image region, a regional mask is generated within the corresponding image region. For the outer boundary of a preset width of an image region, a region boundary mask is generated, wherein the region boundary mask smoothly transitions from the mask value corresponding to the mask within the region to zero; Based on the area mask, the area boundary mask, and the area exterior mask, a region mask matrix is ​​obtained, where the mask value of the area exterior mask is zero.

[0032] Specifically, for the image region specified by the user, when describing the region, since there may be detailed information about important features (texture, color, local shape) at the region boundary, a smooth transition is made at the region boundary, rather than the high value inside the boundary directly becoming the low value outside the boundary, so that the description within the region is more accurate.

[0033] Specifically, the region boundaries are smoothed using Gaussian numbers. For example, blur = GaussianBlur(kernel_size=(5,5), sigma=(0.6, 0.6)), the blur kernel size is (5,5), the standard deviations in the X and Y directions are denoted as (0.6, 0.6), and the region mask matrix (i.e. the smoothing mask) is soft_mask = blur(mask).

[0034] Among them, hard masks completely ignore boundary pixels, while smooth masks can retain some of the boundary weights. Smooth masks transition the boundary to continuous values, resulting in smooth gradient changes and making the model more likely to converge during training.

[0035] Step S203: When performing feature encoding on the image to be described, attention weighting is applied to the features of the image region based on the region mask matrix to obtain the image feature vector.

[0036] Similar to attention-based image coding, the image to be described is feature-encoded. When it enters the attention module, the region mask matrix is ​​fused into the attention module, so that the region mask matrix can assign higher attention weights to the features within the region, realizing attention weighting, and finally outputting the image feature vector.

[0037] Specifically, an adaptive mask is constructed to guide the model to focus on user-specified image regions (such as given bounding boxes). The input image I is processed by a visual encoder to generate a feature map F. Region correlations are calculated using a region attention module. The mask M is then applied to the attention matrix, causing the model to assign higher weights to features within the regions and suppress features outside the regions.

[0038] The attention matrix formula is expressed as: Attn region = Softmax( (Q*Mr)K T / V Where Q, K, and V are the matrices for query, key, and value, respectively, and Mr is the region mask matrix.

[0039] In this region mask matrix, the embedding value within the region is 1 / S, and the embedding value outside the region is 0. S is the number of embedding parameters within the region, i.e., sum(Mr) == 1.

[0040] Step S204: Use the text encoder in the trained visual language model to describe the image feature vector and obtain the content description text.

[0041] The trained visual language model is trained using a loss constructed based on the differences between the full-image description and the region-cropped image description.

[0042] In this embodiment, the visual language model needs to be augmented based on the existing training. During augmentation training, the full image and the cropped region image of the training image need to be described separately, and the loss of the two descriptions is used for augmentation training to make the difference between the full image description and the region description smaller. In other words, the visual language model trained by augmentation training can better suppress the influence of the full image except for the specified region.

[0043] When the trained visual language model is used for description, the content description text obtained is not much different from the description result for image regions, which proves that the trained model can further improve the focusing ability.

[0044] This application embodiment assigns better weights to features within a region through a region attention mask, thereby obtaining a focused image feature vector. The model is trained based on comparing the differences between the full-image description and the region cropped image description, which optimizes the focusing ability of the visual language model, suppresses global interference, and achieves accurate description of the target region.

[0045] See Figure 3 This is a flowchart illustrating an image region content description method provided in Embodiment 3 of this application. Figure 3 As shown, the training process of a well-trained visual language model includes the following steps: Step S301: Obtain the training image and the target region specified by the description of the corresponding training image, as well as the cropped image cropped from the training image based on the target region.

[0046] Step S302: Obtain a pre-trained visual language model, input the training image and target region into the pre-trained visual language model, and output a description of the target region corresponding to the training image as a full-image description.

[0047] Step S303: Input the cropped image into the pre-trained visual language model and output a description of the corresponding cropped image as a region cropping map description.

[0048] Step S304: Obtain the first evaluation result for the full map description and the second evaluation result for the region clipping map description. Based on the difference between the first evaluation result and the second evaluation result, and combined with the preset loss function, obtain the constraint loss.

[0049] Step S305: Optimize the pre-trained visual language model based on the loss and constraint loss described by the generated region clipping map to obtain a trained visual language model.

[0050] Among them, steps S301 to S305 represent the training process of enhancing the visual language model. On the basis of the unchanged architecture of the visual language model, steps S301 and S302 are added to form two kinds of descriptions: one is the full image description (this full image description does not describe the entire image, but describes the target region of the entire image, and is a description obtained with the full image input), and the other is the region cropping image description (this region description is a description of the entire image from the cropping image).

[0051] The evaluation results are obtained by evaluating the two descriptions. The evaluation results are used to construct a new constraint loss. The goal of this constraint loss is to make the model's description of the target region of the whole image similar to the description of the region clipping image, so that the model can focus on the target region of the whole image and suppress other regions.

[0052] Of course, in order to avoid significant changes in the model during training, the loss of the original generated region clipping map description (i.e. the description loss that is actually needed) is used as the basis and together with the constraint loss to optimize the model to obtain a well-trained visual language model.

[0053] Optionally, obtain a first evaluation result for the full map description and a second evaluation result for the region-cropped map description, including: Obtain the first raw prediction value of the full-image description of all training images, and the second raw prediction value of the cropped image description of all cropped images. The logarithmic probability distribution of the first original predicted value is calculated to obtain the first distribution result. All training images are traversed to obtain the first distribution result corresponding to all training images. The average of all first distribution results is calculated to obtain the first evaluation result. The second original predicted value is calculated using a logarithmic probability distribution to obtain the second distribution result. All cropped images are traversed to obtain the second distribution result corresponding to all cropped images. The average of all second distribution results is then calculated to obtain the second evaluation result.

[0054] During the training process, training images are fed into the model for training. The model performs forward inference to calculate the Logit for two descriptions, which reflects the model's confidence in the description. Then, log_softmax is taken as the score, i.e., Logratios = log_softmax(logit).mean().

[0055] Optionally, based on the difference between the first and second evaluation results, and combined with a preset loss function, a constraint loss is obtained, including: The difference between the second evaluation result and the first evaluation result is used to obtain the first difference result. Based on the first difference result, combined with the preset negative logarithm function of sigmoid, the constraint loss is calculated.

[0056] The score for the region-cropped map description is denoted as pi_logratios, and the score for the region description (full map description) under the full map condition is denoted as ref_logratios. The difference between the two description scores is calculated as delta_logratios = pi_logratios − ref_logratios. The constraint loss is constructed based on this difference, using a sigmoid function to construct the contrastive constraint loss. L GDS =−F.logsigmoid(β⋅delta_logratios); In the formula, β is a temperature coefficient used to control the smoothness of the loss and the convergence speed.

[0057] In optimizing target training, minimize L GDS This guides the model to reduce the overall map description score and increase the regional description score, making the output more closely match the content of the target region.

[0058] In addition, the definition of total loss is: L = L caption + λ⋅L GDS; Where Lcaption is the standard image description loss (such as cross-entropy loss), and only the region cropping map description loss is calculated, while the full map region description loss does not calculate Lcaption. λ is the weighting coefficient that balances the two losses.

[0059] The embodiments of this application effectively suppress global information interference unrelated to the target region through a global description suppression mechanism, further improving the model's focusing ability and making the model's recognition accuracy higher.

[0060] See Figure 4 This is a flowchart illustrating an image region content description method provided in Embodiment 4 of this application. Figure 4As shown, before obtaining the first evaluation result for the full map description and the second evaluation result for the region clipping map description in step S304 above, the following steps may also be included: Step S401: Calculate the semantic matching degree between the full image description and the corresponding training image to obtain the full image conditional matching degree.

[0061] Step S402: Calculate the semantic matching degree between the region cropping map description and the corresponding cropped image to obtain the region cropping matching degree.

[0062] Step S403: Based on the full-image condition matching degree and the region cropping matching degree, all training images and cropped images are filtered to obtain the filtered training images and the filtered cropped images.

[0063] The steps S304 above, which involve obtaining the first evaluation result for the full map description and the second evaluation result for the region clipping map description, may include the following steps: Step S404: Obtain the first evaluation result of the full-image description corresponding to the selected training images, and the second evaluation result of the region cropping image description corresponding to the selected cropped images.

[0064] Specifically, for the training image and its designated region (determined by bounding box annotations), two types of descriptions are generated: full-image conditional description (denoted as T g): after the model observes the training image, it answers "what the designated region contains"; region cropping description (denoted as T r): the training image is first cropped into an image I r containing only the designated region, and then the model directly describes the content of I r.

[0065] Extracting image and corresponding descriptive features, calculating semantic matching degree, and filtering out samples that are easily confused by the model are divided into three steps: Calculate semantic matching degree: Specifically, the CLIP model can be used to calculate the semantic matching degree between the text and the corresponding image.

[0066] Full-image conditional matching degree: sg=CLIPSim(I, T g); Region cropping matching degree: sr=CLIPSim(I r, Tr); Calculate semantic difference: The difference between the two matching degrees is calculated by absolute value, using the formula: Δs = sg - sr.

[0067] Screening "region confusion samples": Set a threshold τ. When Δs>τ, it indicates that there is a significant deviation between the full-image condition description and the region clipping description. This sample is identified as a "region confusion sample" and used for subsequent reinforcement training. τ can be set to 0.3-0.4.

[0068] For example, the following data format can be used to process training images: { "image_id": "000134.jpg", # Original image "conversations": [ { "from": "human", "value": "Given the background of the entire image, what do you think is in this specified area [293,607,535,919]?" }, { "from": "gpt", "value": "****." # Corresponds to the description of the entire image area. }, { "from": "human", "value": "What does this region [293,607,535,919] contain against the background of the entire image?" }, { "from": "gpt", "value": "****." # Description of the corresponding cropping area }, ] } Optionally, based on the full-image conditional matching degree and the region cropping matching degree, all training images and cropped images are filtered to obtain the filtered training images and the filtered cropped images, including: For any training image and its corresponding cropped image, the difference between the full-image conditional matching degree of the training image and the region cropping matching degree of the cropped image is obtained to get the second difference result. The absolute value of the second difference result is compared with the preset difference threshold to obtain the comparison result. If the comparison result is that the absolute value of the second difference result exceeds the preset difference threshold, the training image is determined to be the selected training image and the cropped image is determined to be the selected cropped image. Otherwise, the training image and the cropped image are removed.

[0069] Among these methods, training images are filtered by setting a threshold, which accurately and quickly yields the filtering results for more accurate model training.

[0070] This application's embodiments focus on "regional confusion samples" to ensure that training samples are concentrated in scenarios where the model is prone to errors, avoiding interference from invalid samples, thereby efficiently improving the model's perception accuracy and stability for a specified region.

[0071] Corresponding to the image region content description method in the above embodiments, Figure 5 A schematic diagram of the image region content description device provided in Embodiment 5 of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0072] See Figure 5 The image region content description device includes: The data acquisition module 51 is used to acquire the image to be described and the image region specified by the description of the image to be described; Region masking module 52 is used to generate a region mask matrix by combining the attention module of the visual encoder in the trained visual language model with the image regions. The feature encoding module 53 is used to perform attention weighting on the features of the image region based on the region mask matrix when performing feature encoding on the image to be described, so as to obtain the image feature vector. The content description module 54 is used to describe the image feature vector using the text encoder in the trained visual language model to obtain the content description text. The trained visual language model is trained based on the loss constructed by comparing the differences between the full image description and the region cropped image description.

[0073] Optionally, the training process of the trained visual language model includes: The data preparation module is used to obtain training images and the target regions specified by the descriptions of the corresponding training images, as well as cropped images cropped from the training images based on the target regions; The full-image description module is used to obtain a pre-trained visual language model. The training image and the target region are input into the pre-trained visual language model, and the output is a description of the target region of the corresponding training image as a full-image description. The region description module is used to input the cropped image into the pre-trained visual language model and output a description of the corresponding cropped image as a region cropping map description. The loss calculation module is used to obtain the first evaluation result of the full map description and the second evaluation result of the regional clipping map description. Based on the difference between the first evaluation result and the second evaluation result, and combined with the preset loss function, the constraint loss is obtained. The training module is used to optimize the pre-trained visual language model based on the loss and constraint loss described by the generated region clipping map, so as to obtain a trained visual language model.

[0074] Optionally, the loss calculation module includes: The prediction value acquisition unit is used to acquire the first original prediction value of the output full map description corresponding to all training images, and the second original prediction value of the output region cropping map description corresponding to all cropped images. The first evaluation unit is used to calculate the logarithmic probability distribution of the first original predicted value, obtain the first distribution result, traverse all training images, obtain the first distribution result corresponding to all training images, and average all the first distribution results to obtain the first evaluation result. The second evaluation unit is used to calculate the logarithmic probability distribution of the second original predicted value to obtain the second distribution result, traverse all cropped images to obtain the second distribution result corresponding to all cropped images, and average all the second distribution results to obtain the second evaluation result.

[0075] Optionally, the loss calculation module includes: The first difference calculation unit is used to calculate the difference between the second evaluation result and the first evaluation result to obtain the first difference result; The loss calculation unit is used to calculate the constraint loss based on the first difference result and in combination with the preset negative logarithm function of sigmoid.

[0076] Optionally, the above training process also includes: The full-image matching module is used to calculate the semantic matching degree between the full-image description and the corresponding training image before obtaining the first evaluation result of the full-image description and the second evaluation result of the region cropping image description, so as to obtain the full-image conditional matching degree. The region matching module is used to calculate the semantic matching degree between the region cropping map description and the corresponding cropped image, and obtain the region cropping matching degree. The image filtering module is used to filter all training images and cropped images based on the full-image condition matching degree and the region cropping matching degree, so as to obtain the filtered training images and the filtered cropped images. The data preparation module includes: The data preparation unit is used to obtain the first evaluation result of the full-image description corresponding to the selected training images, and the second evaluation result of the region cropping image description corresponding to the selected cropped images.

[0077] Optionally, the image filtering module includes: The second difference calculation unit is used to calculate the difference between the full-image conditional matching degree of the training image and the region cropping matching degree of the cropping image for any training image and the corresponding cropped image, and obtain the second difference result. The image filtering unit is used to compare the absolute value of the second difference result with a preset difference threshold to obtain a comparison result. If the comparison result is that the absolute value of the second difference result exceeds the preset difference threshold, then the training image is determined to be the selected training image and the cropped image is the selected cropped image; otherwise, the training image and the cropped image are discarded.

[0078] Optionally, the region mask module 52 includes: The intra-region masking unit is used to generate intra-region masks within the corresponding image region by combining the attention module of the visual encoder in the trained visual language model with the image region. The region boundary mask unit is used to generate a region boundary mask for the outer boundary of a preset width of an image region, wherein the region boundary mask smoothly transitions from the mask value corresponding to the mask within the region to zero; The region mask matrix unit is used to obtain the region mask matrix based on the region mask, the region boundary mask, and the region exterior mask, where the mask value of the region exterior mask is zero.

[0079] It should be noted that the information interaction and execution process between the above modules, units, and sub-units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0080] Figure 6 This is a schematic diagram of the structure of a computer device provided in Embodiment Six of this application. Figure 6 As shown, the computer device of this embodiment includes: at least one processor ( Figure 6 The image region content description method embodiments described above are shown in only one (i.e., a memory and a computer program stored in the memory and executable on at least one processor, wherein the processor executes the computer program to implement the steps of any of the above-described image region content description method embodiments).

[0081] This computer device may include, but is not limited to, a processor, memory, and a database. Those skilled in the art will understand that... Figure 6 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the figures, or combinations of certain components, or different components. For example, they may also include databases, network interfaces, displays, and input devices, which can be connected via a data bus.

[0082] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0083] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0085] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a computer device, it enables the computer device to execute the steps in the above method embodiments.

[0086] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0087] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0088] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for describing the content of an image region, characterized in that, include: Obtain the image to be described and the image region specified by the description of the image to be described; Using the attention module of the visual encoder in the trained visual language model, combined with the image region, a region mask matrix is ​​generated; When performing feature encoding on the image to be described, attention weighting is applied to the features of the image region based on the region mask matrix to obtain the image feature vector; The image feature vector is described using the text encoder in the trained visual language model to obtain the content description text. The trained visual language model is trained using a loss constructed based on the difference between the full image description and the region cropped image description.

2. The image region content description method according to claim 1, characterized in that, The training process of the trained visual language model includes: Obtain a training image and a target region specified by a description corresponding to the training image, as well as a cropped image cropped from the training image based on the target region; Obtain a pre-trained visual language model, input the training image and the target region into the pre-trained visual language model, and output a description of the target region corresponding to the training image as a full-image description; The cropped image is input into the pre-trained visual language model, and the output is a description of the cropped image corresponding to the region cropping map description; Obtain a first evaluation result for the description of the full map and a second evaluation result for the description of the cropped map of the region. Based on the difference between the first evaluation result and the second evaluation result, and combined with a preset loss function, obtain the constraint loss. The pre-trained visual language model is optimized based on the loss described in the generation of the region clipping map and the constraint loss to obtain a trained visual language model.

3. The image region content description method according to claim 2, characterized in that, The process of obtaining a first evaluation result for the full-map description and a second evaluation result for the region-cropped map description includes: Obtain the first original predicted value of the full image description corresponding to all training images, and the second original predicted value of the region cropping image description corresponding to all cropped images; The logarithmic probability distribution of the first original predicted value is calculated to obtain the first distribution result. All training images are traversed to obtain the first distribution result corresponding to all training images. The average of all first distribution results is calculated to obtain the first evaluation result. The second original predicted value is calculated using a logarithmic probability distribution to obtain a second distribution result. All cropped images are traversed to obtain the second distribution result corresponding to all cropped images. The average of all second distribution results is then calculated to obtain a second evaluation result.

4. The image region content description method according to claim 2, characterized in that, The step of obtaining the constraint loss based on the difference between the first evaluation result and the second evaluation result, combined with a preset loss function, includes: The difference between the second evaluation result and the first evaluation result is obtained to obtain the first difference result; Based on the first difference result, and combined with the preset negative logarithm function of sigmoid, the constraint loss is calculated.

5. The image region content description method according to claim 2, characterized in that, Before obtaining the first evaluation result for the full map description and the second evaluation result for the region clipping map description, the method further includes: Calculate the semantic matching degree between the full-image description and the corresponding training image to obtain the full-image conditional matching degree; Calculate the semantic matching degree between the region cropping map description and the corresponding cropped image to obtain the region cropping matching degree; Based on the full-image condition matching degree and the region cropping matching degree, all training images and cropped images are filtered to obtain the filtered training images and the filtered cropped images. The process of obtaining a first evaluation result for the full-map description and a second evaluation result for the region-cropped map description includes: Obtain a first evaluation result for the full-image description corresponding to the selected training image, and a second evaluation result for the region cropping image description corresponding to the selected cropped image.

6. The image region content description method according to claim 5, characterized in that, The step of filtering all training images and cropped images based on the full-image conditional matching degree and the region cropping matching degree to obtain the filtered training images and the filtered cropped images includes: For any training image and its corresponding cropped image, the full-image conditional matching degree of the training image is subtracted from the region cropping matching degree of the cropped image to obtain a second difference result. The absolute value of the second difference result is compared with a preset difference threshold to obtain a comparison result. If the comparison result is that the absolute value of the second difference result exceeds the preset difference threshold, then the training image is determined to be a selected training image and the cropped image is a selected cropped image; otherwise, the training image and the cropped image are discarded.

7. The image region content description method according to any one of claims 1 to 6, characterized in that, The step of using the attention module of the visual encoder in the trained visual language model, combined with the image region, to generate a region mask matrix includes: Using the attention module of the visual encoder in the trained visual language model, combined with the image region, a regional mask corresponding to the image region is generated. For the outer boundary of the image region with a preset width, a region boundary mask is generated, wherein the region boundary mask smoothly transitions from the mask value corresponding to the mask within the region to zero; A region mask matrix is ​​obtained based on the region mask, the region boundary mask, and the region exterior mask, wherein the mask value of the region exterior mask is zero.

8. An image region content description device, characterized in that, include: The data acquisition module is used to acquire the image to be described and the image region specified by the description of the image to be described; The region masking module is used to generate a region mask matrix by combining the image regions with the attention module of the visual encoder in the trained visual language model. The feature encoding module is used to perform attention weighting on the features of the image region based on the region mask matrix when performing feature encoding on the image to be described, so as to obtain an image feature vector; The content description module is used to describe the image feature vector using the text encoder in the trained visual language model to obtain content description text. The trained visual language model is trained based on a loss constructed by comparing the differences between the full image description and the region cropped image description.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image region content description method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image region content description method as described in any one of claims 1 to 7.

Images