Image description information generation method and device, computer program product and equipment

Abstract

The invention relates to the technical field of image processing, and relates to an image description information generation method and device, a computer program product and electronic equipment. The image description information generation method comprises the steps of obtaining a target image; extracting global image features of the target image, and generating global description information according to the global image features; local image features of a target object in the target image are extracted, local description information is generated according to the local image features, the target object is determined based on an object detection result of the target image, and the local description information at least comprises object feature information and position information of the target object; and generating structured description text information according to the global description information and the local description information. The quality of the generated image description information can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and more specifically, to a method for generating image description information, a device for generating image description information, a computer program product, and an electronic device. Background Art

[0002] With the development of image processing technology, generating images through text language description is one of the important directions of modern artificial intelligence in the field of computer vision, and it also brings convenience to users who need to create images. At present, images can be generated by expanding or polishing the received text language description. However, the low accuracy of text language description affects the quality of image generation and affects the user experience to a certain extent.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method and device for generating image description information, a computer program product and an electronic device, thereby improving the quality of generated image description information at least to a certain extent.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a method for generating image description information is provided, comprising: acquiring a target image; extracting global image features of the target image, and generating global description information based on the global image features; extracting local image features of a target object in the target image, and generating local description information based on the local image features, wherein the target object is determined based on an object detection result of the target image, and the local description information includes at least object feature information and position information of the target object; and generating structured description text information based on the global description information and the local description information.

[0007] In an exemplary embodiment of the present disclosure, the method further includes: receiving instruction description information of the target image, and extracting text features of the instruction description information; extracting global image features of the target image includes: determining the global image features based on image fusion features extracted from the target image; or, determining the global image features by combining the image fusion features and the text features.

[0008] In an exemplary embodiment of the present disclosure, if the global image feature is determined based on the image fusion feature extracted from the target image, the method includes: dividing the target image into image blocks, and performing feature encoding on each of the image blocks to obtain image block features; determining initial image features based on the features of each image block; performing denoising processing on the initial image features several times through a diffusion model, and fusing the intermediate image features corresponding to each denoising processing to obtain the image fusion feature; and determining the global image feature based on the image fusion feature.

[0009] In an exemplary embodiment of the present disclosure, if the image fusion feature and the text feature are combined to determine the global image feature, the method includes: aligning the image fusion feature and the text feature; based on the attention mechanism, according to the feature correlation between the image fusion feature and the text feature after feature alignment, fusing the image fusion feature and the text feature after feature alignment to obtain the global image feature.

[0010] In an exemplary embodiment of the present disclosure, the method also includes: performing object detection on the target image to determine the initial object in the target image; extracting a keyword vector from the instruction description information; determining a reference object corresponding to the keyword vector from the initial object based on the correlation between the global image features and the keyword vector; obtaining a correlation score between the reference object and the keyword vector, and determining the target object from the reference object based on the correlation score.

[0011] In an exemplary embodiment of the present disclosure, the global description information and local description information of the target image are generated using an image description information extraction model; wherein the process of obtaining the image description information extraction model includes: obtaining training samples, the training samples include sample images, global reference descriptions and local reference descriptions, the local reference descriptions include at least reference object feature information and reference position information of the reference object; and using the training samples to train the target model to obtain the image description information extraction model.

[0012] In an exemplary embodiment of the present disclosure, the method of training a target model using the training samples to obtain the image description information extraction model includes: inputting the sample image into the image description information extraction model to obtain a global generated description and a local generated description; constructing a text generation loss based on the global generated description and the global reference description; constructing a structured format loss based on the reference position information in the local generated description and the local reference description; the reference position information includes at least global positioning information, local positioning information, global area range and image depth information of the reference object; constructing a feature matching loss based on the reference object feature information in the local generated description and the local reference description; and adjusting the parameters in the target model in combination with the text generation loss, the structured format loss and the feature matching loss to obtain the image description information extraction model.

[0013] In an exemplary embodiment of the present disclosure, constructing a structured format loss based on the reference position information in the local generated description and the local reference description includes: determining a position loss based on the predicted global positioning in the local generated description and the global positioning information; determining an offset loss based on the predicted local positioning in the local generated description and the local positioning information; constructing a regional range loss based on the predicted global regional range in the local generated description and the global regional range; constructing a depth loss based on the predicted depth value in the local generated description and the image depth information; and determining the structured format loss by combining the position loss, the offset loss, the regional range loss and the depth loss.

[0014] In an exemplary embodiment of the present disclosure, determining the global positioning information and the local positioning information of the reference object includes: dividing the sample image into reference image blocks, and determining a plurality of candidate global positioning information according to the reference image blocks; dividing the reference image blocks into image sub-blocks, and determining a plurality of candidate local positioning information according to the positions of the image sub-blocks in the reference image blocks; acquiring the global positioning information corresponding to the reference object from the plurality of candidate global positioning information, and acquiring the local positioning information corresponding to the reference object from the plurality of candidate local positioning information.

[0015] In an exemplary embodiment of the present disclosure, determining the global area range of the reference object includes: determining a target center based on the global positioning information and the local positioning information; obtaining a target area occupied by the reference object in the sample image; the target area includes at least an area size and an area type; taking the target center as the center point, determining the global area range of the reference object based on the target center and the target area.

[0016] In an exemplary embodiment of the present disclosure, the acquiring of training samples further includes: performing image enhancement on the sample image, and performing text enhancement on the global reference description and the local reference description respectively.

[0017] In an exemplary embodiment of the present disclosure, the obtaining of training samples further includes: dividing the global reference description into multiple sub-descriptions, and dividing the local reference description into multiple sub-descriptions; wherein the length of each sub-description is less than a preset length and is semantically complete.

[0018] In an exemplary embodiment of the present disclosure, the method further includes: responding to a description adjustment operation, and adjusting the description text in the structured description text information according to the description adjustment operation.

[0019] According to one aspect of the present disclosure, there is provided a device for generating image description information, comprising: an image input module for acquiring a target image; a first image processing module for extracting global image features of the target image and generating global description information based on the global image features; a second image processing module for extracting local image features of a target object in the target image and generating local description information based on the local image features, wherein the target object is determined based on an object detection result of the target image, and the local description information includes at least object feature information and position information of the target object; and a third image processing module for generating structured description text information based on the global description information and the local description information.

[0020] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, any one of the methods described above is implemented.

[0021] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods by executing the executable instructions.

[0022] The method for generating image description information in the exemplary embodiment of the present disclosure, on the one hand, acquires a target image, extracts global image features of the target image, generates global description information based on the global image features, extracts local image features of the target object in the target image, and generates local description information based on the local image features. This process generates local description information for the target object in the target image while generating global description information for the target image as a whole. The relative positions between the target objects can be clearly defined through the position information in the local description information, thereby achieving a refined description of the target image and providing high-quality information for guiding the image generation model. On the other hand, the global description information and the local description information are separated to make the description information clear and organized, and structured description text information is generated based on the global description information and the local description information, thereby increasing the clarity and readability of the description information, making it easy to modify, and improving the flexibility of generating image description information.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown in an exemplary and non-limiting manner.

[0025] Figure 1 A schematic diagram of an application environment according to an exemplary embodiment of the present disclosure is shown.

[0026] Figure 2 A flowchart of a method for generating image description information according to an exemplary embodiment of the present disclosure is shown.

[0027] Figure 3 A flow chart of determining global image features based on image fusion features extracted from a target image according to an exemplary embodiment of the present disclosure is shown.

[0028] Figure 4 A flow chart of combining image fusion features and text features to determine global image features according to an exemplary embodiment of the present disclosure is shown.

[0029] Figure 5 A flowchart of an implementation method for determining a target object in a target image according to an exemplary embodiment of the present disclosure is shown.

[0030] Figure 6 A flowchart of a model training method according to an exemplary embodiment of the present disclosure is shown.

[0031] Figure 7A flowchart of using training samples to train a target model to obtain an image description information extraction model according to an exemplary embodiment of the present disclosure is shown.

[0032] Figure 8 A flowchart of constructing structured format loss according to an exemplary embodiment of the present disclosure is shown.

[0033] Fig. 9 A schematic diagram of determining global positioning information and local positioning information of a reference object according to an exemplary embodiment of the present disclosure is shown.

[0034] Fig.10 A schematic diagram of a target image according to an exemplary embodiment of the present disclosure is shown.

[0035] Fig.11 A schematic diagram of structured description text information of a target image according to an exemplary embodiment of the present disclosure is shown.

[0036] Fig.12 A schematic diagram showing the composition of a device for generating image description information according to an exemplary embodiment of the present disclosure is shown.

[0037] Fig.13 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0038] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0039] The exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and the concepts of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their detailed description will be omitted.

[0040] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0042] The method for generating image description information provided by the exemplary embodiment of the present disclosure can be applied to Figure 1 In the application environment shown, the terminal 101 communicates with the server 102 via a network. The data storage system can store data that the server 102 needs to process. The data storage system can be integrated on the server 102, or placed on the cloud or other network servers.

[0043] In an exemplary embodiment, the generation of the image description information provided by the exemplary embodiment of the present disclosure may be performed by the server 102, and the corresponding image description information generation device is set in the server 102. Correspondingly, in this mode of execution by the server 102, the server 102 may start to execute the steps in the technical solution of the exemplary embodiment of the present disclosure in response to a trigger command, wherein the trigger command may be sent by a terminal used by a user, or may be triggered locally by the server in response to some automation events.

[0044] The server 102 can obtain a target image, extract global image features of the target image, and generate global description information based on the global image features; then extract local image features of a target object in the target image, and generate local description information based on the local image features, wherein the target object is determined based on an object detection result of the target image, and the local description information includes at least object feature information and location information of the target object; finally, structured description text information is generated based on the global description information and the local description information.

[0045] The server 102 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server 102 may perform background tasks.

[0046] Furthermore, in another exemplary embodiment, the terminal 101 may also have similar functions to the server 102 , so as to execute the method for generating image description information provided by the exemplary embodiment of the present disclosure.

[0047] The terminal 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an IoT device, and a portable wearable device. The IoT device may be a smart speaker, a smart TV, a smart air conditioner, and a smart vehicle-mounted device, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The terminal 101 may also be referred to as a mobile terminal, a terminal device, a mobile device, etc. The exemplary embodiments of the present disclosure do not limit the type of the terminal 101.

[0048] In addition, the technical solution of the exemplary embodiment of the present disclosure can also be executed by the terminal 101 and the server 102 in collaboration. In this way of collaborative execution by the terminal 101 and the server 102, some steps in the technical solution provided by the exemplary embodiment of the present disclosure are executed by the terminal 101, and other steps are executed by the server 102.

[0049] It should be noted that in this mode of collaborative execution by the terminal 101 and the server 102, the steps respectively executed by the terminal 101 and the server 102 can be dynamically adjusted according to actual conditions, and there is no special restriction on this.

[0050] The terminal 101 and the server 102 may be directly or indirectly connected via wireless communication, and the exemplary embodiments of the present disclosure do not impose any special limitation thereto.

[0051] The use of artificial intelligence to generate images has high requirements for text language descriptions. Once the user enters the text language description, he or she will not be able to intervene in the process of artificial intelligence generating images. To further improve the quality of generated images, the computer's "image description" function, that is, image description, can be realized through algorithms, which converts the content information in the image into image description information, and then uses the image description information to guide the image generation process.

[0052] However, in the current solution, images are generated by expanding or polishing the received text language description. However, the low accuracy of the text language description affects the quality of image generation, and a detailed description of the image cannot be achieved, resulting in information loss or confusion in the image generation process, which affects the user experience to a certain extent.

[0053] In an exemplary embodiment of the present disclosure, a method for generating image description information is provided to solve one or more of the above problems. Figure 2 FIG. 1 is a flow chart of a method for generating image description information according to an exemplary embodiment of the present disclosure. Figure 2 As shown, the generation method includes steps S210 to S240, which are specifically as follows:

[0054] In step S210, a target image is acquired.

[0055] In an exemplary embodiment of the present disclosure, the target image is an image to be processed for generating image description information. The target image may be a locally stored image, or the target image may be an image acquired in real time based on a user-specified operation, for example, the target image may be an image acquired in real time by a user based on a screenshot operation, or may be an image on a terminal screen acquired in real time when a user triggers the generation of image description information by long pressing a specified area of ​​the screen, or may be an image acquired in real time based on an image acquisition component of the terminal. The exemplary embodiments of the present disclosure do not limit the method for acquiring the target image.

[0056] In some optional embodiments, the target image may be preprocessed, such as format conversion, image enhancement, etc. For example, the target image may be converted into a target image format, such as JPEG, PNG, TIFF, etc., to ensure universality and compatibility. Image enhancement includes but is not limited to normalization, rotation, scaling, cropping, etc., which can be selected according to actual needs and are not limited to this.

[0057] In step S220, global image features of the target image are extracted, and global description information is generated according to the global image features.

[0058] In an exemplary embodiment of the present disclosure, the description information of an image refers to a natural language description of the corresponding image obtained by taking the image as input through models and calculations. The description information of the image can be a description of the properties of each component in the image, or a description of the relationship between its various components. Among them, the global image features include global and local features of the target image, which comprehensively describe the content of the target image. Global description information can be generated by converting global image features into text descriptions. Among them, the target model can be trained to make the target model understand the relationship between image features and text descriptions, so as to generate global description information based on the global image features using the trained target model.

[0059] Exemplarily, the target model can be constructed using a lightweight neural network, such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Networks), DNN (Deep Neural Networks), etc. Relative to lightweight neural networks, a target model with a more complex network structure can also be used, for example, a heavyweight neural network can be constructed, such as Variational Autoencoders (VAE), BERT (Bidirectional Encoder Representation from Transformers, Bidirectional Transformers Feature Encoder), Transformer (a language processing model), and any heavyweight neural network that can be used for feature extraction. Furthermore, the target model can also be a large model of 100 million levels, such as a deep learning model trained using a large amount of data. The exemplary embodiments of the present disclosure are described by taking the target model as a large language model as an example.

[0060] In step S230, local image features of the target object in the target image are extracted, and local description information is generated according to the local image features.

[0061] In an exemplary embodiment of the present disclosure, the target object is determined based on the object detection result of the target image. In feature extraction and generation of description information, the target object needs to be determined, so as to extract local image features for the target object, fully exploit key local image features, and improve the effectiveness of feature extraction. The object detection model can be used to perform object detection on the target image, and the target object can be determined based on the object detection result. Optionally, if the instruction description information of the target image is obtained at the same time, the process of determining the target object can also be based on the guidance of the instruction description information, which will be described in detail later.

[0062] The local image features are features of the target object in the corresponding target image. The local image features of different target objects can be extracted respectively. The local description information includes at least object feature information and position information of the target object. The object feature information includes style, color, atmosphere, object features, etc., which are not limited. The position information indicates the relevant information describing the position of the target object in the target image. Based on the position information of different target objects, the relative position relationship between the different target objects can be obtained.

[0063] The local description information can also be generated by converting the local image features into text descriptions. The target model can be trained to understand the relationship between the image features and the text descriptions, so that the local description information can be generated according to the local image features using the trained target model.

[0064] In step S240, structured description text information is generated according to the global description information and the local description information.

[0065] In an exemplary embodiment of the present disclosure, structured description text information refers to a detailed description of all target objects in a target image, including global description information and local description information. The global description information and the local description information are unified into a structured format, and the global description information and the local description information are separated to make the structured description text information more organized and clear.

[0066] For example, the global description information and the local description information may be converted into a Canvas structured description, an SVG structured description, etc., which may be selected according to actual needs and is not limited thereto.

[0067] The method for generating image description information in the exemplary embodiment of the present disclosure, on the one hand, acquires a target image, extracts global image features of the target image, generates global description information based on the global image features, extracts local image features of the target object in the target image, and generates local description information based on the local image features. This process generates local description information for the target object in the target image while generating global description information for the target image as a whole. The relative positions between the target objects can be clearly defined through the position information in the local description information, thereby achieving a refined description of the target image and providing high-quality information for guiding the image generation model. On the other hand, the global description information and the local description information are separated to make the description information clear and organized, and structured description text information is generated based on the global description information and the local description information, thereby increasing the clarity and readability of the description information, making it easy to modify, and improving the flexibility of generating image description information.

[0068] In an exemplary embodiment, instruction description information of the target image may also be received, and text features of the instruction description information may be extracted.

[0069] The instruction description information may be a natural language description of the instruction processing requirements provided by the user, such as "help me generate an image description information with elements of a beauty and a mecha tank."

[0070] Based on this, a method for extracting global image features is provided. Extracting global image features of a target image includes:

[0071] The global image feature is determined based on the image fusion feature extracted from the target image; or, the global image feature is determined by combining the image fusion feature and the text feature.

[0072] Specifically, in the scenario where the user only provides the target image, the global image features can be determined based on the image fusion features extracted from the target image. In the scenario where the user provides the target image and its instruction description information, the global image features can be determined by combining the image fusion features and text features.

[0073] The text features of the extracted instruction description information may include:

[0074] First, the instruction description information is segmented to obtain several words or phrases. Then, the segmented result is converted into an embedding vector through a pre-trained language model. The embedded vector is parsed to obtain a structured requirement vector, namely, a text feature.

[0075] Among them, the embedding vector indicates the semantic information of the instruction description information, and the embedding vector is parsed to determine the key objects in the instruction description information, such as "beauty", "mech tank", "canvas composition", etc., to determine the text features including the key objects in the instruction description information and their associations.

[0076] In an exemplary embodiment, if Figure 3 As shown, if the global image features are determined based on the image fusion features extracted from the target image, steps S310 to S340 may be included:

[0077] Step S310, dividing the target image into image blocks, and performing feature encoding on each image block to obtain image block features.

[0078] The target image is divided into image blocks of fixed size. For example, a target image of 224x224 pixels is divided into image blocks of 16x16 pixels, a total of 196 image blocks, each of which is processed as a basic unit.

[0079] Before dividing the target image into image blocks, the target image may also be preprocessed, such as normalization, resizing, etc. In some optional embodiments, the pixel values ​​of the target image may be normalized to the range of [0, 1] or [-1, 1] to standardize the target image and ensure the stability of the subsequent processing. Optionally, the target image may be adjusted to a fixed size (e.g., 224x224 pixels) to facilitate subsequent processing by unifying the size of the target image. It should also be understood that the exemplary embodiments of the present disclosure may also select a processing method for the target image according to actual needs, without limitation thereto.

[0080] In some optional embodiments, a linear layer may be used to perform feature encoding on each image block. Exemplarily, each 16×16 image block may be flattened by a linear layer and mapped to a high-dimensional feature space. For example, a 16×16 image block is flattened into a 256-dimensional feature vector to obtain the image block feature of the image block.

[0081] Step S320: determining initial image features according to features of each image block.

[0082] The image block features of each image block corresponding to the target image are fused to obtain an initial image feature, which represents a rough feature of the target image. The fusion method is, for example, splicing, which is not limited.

[0083] Step S330: Perform several noise reduction processes on the initial image features through the diffusion model, and fuse the intermediate image features corresponding to each noise reduction process to obtain image fusion features.

[0084] The diffusion model is used to diffuse the initial image features to reduce image noise, and each step can process the features through the transformer encoder.

[0085] Specifically, the initial image features can be input into the transformer encoder for the first step of feature extraction. The transformer encoder includes a multi-layer attention network and a feedforward neural network, each layer of which captures the relationship and global information between image blocks. After the first step of feature extraction, the features are further refined through the denoising operation, and the refined features are input into the transformer encoder again, and the above process is repeated, and each round of processing obtains the intermediate image features.

[0086] Furthermore, the intermediate image features corresponding to all rounds are fused to obtain image fusion features, which include global and local information of the target image and comprehensively characterize the content of the target image.

[0087] Step S340: determining global image features based on the image fusion features.

[0088] After obtaining the image fusion features, the image fusion features can be used as global image features.

[0089] By dividing the target image into image blocks and using a gradual denoising method through a diffusion process, the noise is gradually reduced from the rough feature state of the target image to a fine state to obtain the global image features. This process gradually approaches the global image features to the target image through a diffusion process, thereby improving the accuracy and refinement of the global image features.

[0090] In an exemplary embodiment, if Figure 4As shown, if the image fusion feature and the text feature are combined to determine the global image feature, the steps S410 and S420 may be included:

[0091] Step S410: aligning the image fusion features and the text features.

[0092] The image fusion features and text features can be aligned on the same dimension to ensure that the two are effectively fused later.

[0093] Step S420: Based on the attention mechanism, according to the feature correlation between the image fusion features after feature alignment and the text features, the image fusion features after feature alignment and the text features are fused to obtain the global image features.

[0094] Based on the multi-head attention mechanism, the correlation between the image fusion features and the text features after feature alignment can be calculated. Since each attention head can focus on different feature dimensions, the output of the multi-head attention mechanism is fused to achieve the fusion of multimodal features and obtain the global image feature. The global image feature includes the content of the target image and the content of the instruction description information.

[0095] By combining the target image and instruction description information to determine the global image features, the extracted global image features can meet the actual needs of the user, thereby improving the accuracy of the generated image description information.

[0096] In an exemplary embodiment, after obtaining the global image features, the global image features can be converted into natural language descriptions through the decoder of the model to obtain global description information. The decoder maps the global image features to text representations based on the attention mechanism to generate semantically consistent global description information. The global description information may include but is not limited to the target object, background, scene, style and atmosphere information of the target image. Similarly, after obtaining the local image features, the local image features can be converted into natural language descriptions through the decoder of the model to obtain local description information. The decoder maps the local image features to text representations based on the attention mechanism to generate semantically consistent local description information. The local description information includes the location information of the target object and object feature information (such as object features, atmosphere, style, etc.).

[0097] In some optional embodiments, after obtaining the global image description and the local image description, the two may be optimized. For example, natural language processing technology may be used to process and polish the generated description to ensure the fluency and coherence of the description.

[0098] In some optional embodiments, the global image description may be segmented into multiple sub-descriptions, and the local image description may be segmented into multiple sub-descriptions, so that each sub-description maintains semantic integrity, thereby improving the accuracy of image processing based on the final generated image description information and avoiding problems such as semantic truncation.

[0099] In an exemplary embodiment, a method for determining a target object in a target image is also provided. Figure 5 As shown, it may include steps S510 to S540:

[0100] Step S510: performing object detection on the target image to determine the initial object in the target image.

[0101] Object detection for a target image can be achieved by using a pre-trained object detection model, wherein the model to be trained is trained using sample pairs of image-reference object labels to obtain an object detection model, which is not specifically limited.

[0102] The object detection model is used to perform object detection on the target image to determine the initial object in the target image according to the detection result.

[0103] Step S520: extracting keyword vectors from instruction description information.

[0104] The instruction description information can be preprocessed by word segmentation, text embedding, etc. to generate instruction semantic vectors, and then extract key objects based on the instruction semantic vectors, and generate keyword vectors corresponding to the key objects. For example, the instruction description information is "help me generate an image description information with beauty and mecha elements", and the extracted keyword vectors are the vector corresponding to "beauty", the vector corresponding to "mecha", and the vector corresponding to "canvas composition".

[0105] Step S530: Determine a reference object corresponding to the keyword vector from the initial objects according to the correlation between the global image feature and the keyword vector.

[0106] First, the global image features and keyword vectors are aligned to ensure that they are processed on the same dimension. Then, the correlation between the global image features and the keyword vector is calculated based on the attention mechanism. Since each attention head focuses on different feature dimensions, the objects and their regions related to the keyword vector in the target image can be identified. Finally, the reference object corresponding to the keyword vector is determined from the initial screening results of the initial objects.

[0107] Step S540: Obtain the relevance score between the reference object and the keyword vector, and determine the target object from the reference object based on the relevance score.

[0108] The relevance score represents the relevance between the reference object and the instruction description information provided by the user. The reference object with a higher relevance score will be regarded as an important element and the focus of the local description. The relevance score between each reference object and the keyword vector can be calculated using a preset relevance algorithm. The preset relevance algorithm can be determined according to the actual situation, for example, a cosine similarity algorithm.

[0109] By detecting objects and calculating feature correlations, we can accurately mine target objects in images, focus on them when making local descriptions, and avoid redundancy in local description information. This helps to highlight the key points of the target image in the final structured description text, meeting user expectations.

[0110] In an exemplary embodiment, the global description information and local description information of the target image are generated using an image description information extraction model. The image description information extraction model is a pre-trained large language model. Figure 6 As shown, a model training method is provided, which specifically includes:

[0111] Step S610: Acquire a training sample, where the training sample includes a sample image, a global reference description, and a local reference description, where the local reference description includes at least reference object feature information and reference position information of a reference object.

[0112] Sample images of various scenes can be obtained from public data sets or with authorization, so that the sample images can cover various scenes, objects and action types, and increase the generalization ability of the model. In addition, global reference descriptions and local reference descriptions are generated for the sample images, so that the contents of the global reference descriptions and local reference descriptions are detailed and structured, including at least reference object features and reference location information. For example, it can include information such as position, offset, area size, relative depth, color, style and atmosphere, which can be flexibly selected according to needs and there is no restriction on this.

[0113] Among them, the global reference description and the local reference description can be automatically generated through the existing image analysis method, and then the global reference description and the local reference description can be corrected by manual correction, including but not limited to correction position, offset, area size and other information. The global reference description and the local reference description can also be supplemented, and there is no restriction on this.

[0114] In an optional embodiment, image enhancement may be performed on the sample image, and text enhancement may be performed on the global reference description and the local reference description respectively.

[0115] Specifically, the image enhancement methods include but are not limited to rotation, scaling, cropping, color adjustment, etc., to increase the generalization ability of the model. The text enhancement methods include but are not limited to synonym replacement, word order adjustment, etc., so as to enrich the description information, enhance the model's ability to understand diverse expressions, and improve the accuracy of the description information generated by the model.

[0116] In an optional embodiment, the global reference description may be divided into multiple sub-descriptions, and the local reference description may be divided into multiple sub-descriptions; wherein the length of each sub-description is less than a preset length and is semantically complete.

[0117] Specifically, in order to avoid truncation and loss of semantics when encoding the image description information when using the generated image description information for image generation, the global reference description can be divided into multiple sub-descriptions, and the local reference description can be divided into multiple sub-descriptions. For example, the length of each sub-description is less than 75 tokens, that is, the global reference description can include multiple sub-description groups with a length of less than 75 tokens, and each local reference description also includes multiple sub-description groups with a length of less than 75 tokens. The length of the division can be set according to actual needs, and there is no specific restriction on this.

[0118] By dividing the global reference description and the local reference description into sub-descriptions, the information in the subsequent text encoding process is ensured to be complete and semantic loss is avoided, thereby ensuring the accuracy and coherence of the generated image.

[0119] Step S620: Use the training samples to train the target model to obtain an image description information extraction model.

[0120] The target model is trained through training samples to obtain an image description information extraction model for generating structured global description information and local description information of the target image.

[0121] In an exemplary embodiment, if Figure 7 As shown, using training samples to train the target model, the image description information extraction model may include:

[0122] Step S710: Input the sample image into the image description information extraction model to obtain the global generated description and the local generated description.

[0123] The sample image is analyzed and predicted through the image description information extraction model to obtain the global generated description and the local generated description.

[0124] Step S720: Construct a text generation loss based on the global generation description and the global reference description.

[0125] The text generation loss is used to evaluate the difference between the global generated description and the global reference description generated by the model. The text generation loss can be expressed by the following formula:

[0126]

[0127] Among them, y t is the tth word in the global reference description, is the predicted concept rate of the tth word in the globally generated description, and T is the total length of the description.

[0128] Step S730: construct a structured format loss according to the reference position information in the local generation description and the local reference description.

[0129] The structured format describes a structured format for constraining the generation of description information. The reference position information at least includes global positioning information, local positioning information, global area range and image depth information of the reference object.

[0130] Specifically, Figure 8 As shown, according to the reference position information in the local generation description and the local reference description, constructing the structured format loss may include:

[0131] Step S810: Determine the position loss based on the predicted global positioning and global positioning information in the local generated description.

[0132] The position loss is used to evaluate the accuracy of the global position of the reference object in the sample image. The position loss can be expressed as follows:

[0133]

[0134] Among them, loc i For global positioning information, To predict global positioning, such as the predicted location probability.

[0135] Step S820: Determine the offset loss based on the predicted local positioning and local positioning information in the local generation description.

[0136] The offset loss is used to evaluate the accuracy of the offset of the reference object in the sample image. The offset loss can be expressed by the following formula:

[0137]

[0138] Among them, offset i is the local positioning information, To predict local positioning, such as the predicted offset probability.

[0139] Step S830: construct a regional range loss according to the predicted global regional range and the global regional range in the local generated description.

[0140] The area range loss is used to evaluate the accuracy of the area size of the reference object. The area range loss can be expressed as follows:

[0141]

[0142] Among them, area i For global zone scope, To predict the global region range, such as the predicted region size probability.

[0143] Step S840: construct a depth loss based on the predicted depth value and image depth information in the local generated description.

[0144] Depth loss is used to evaluate the accuracy of image depth and can be expressed by the following formula:

[0145]

[0146] Among them, L depth is the regression loss (such as mean square error), depth i is the image depth information, is the predicted depth value.

[0147] It should be noted that the formula in step S810 to step S840 is an example of calculating the loss, and the exemplary embodiments of the present disclosure may adjust the calculation formula according to actual needs.

[0148] Step S850: Determine the structured format loss by combining the position loss, the offset loss, the area range loss and the depth loss.

[0149] The position loss, offset loss, area range loss and depth loss may be summed to obtain a structured format loss, or weight information of each loss may be obtained, and the position loss, offset loss, area range loss and depth loss may be summed based on the weight information to obtain a structured format loss.

[0150] An exemplary embodiment of the present disclosure constructs a structured format loss by combining position loss, offset loss, area range loss, and depth loss so that the model learns to generate description information that conforms to a structured format.

[0151] Step S740: construct a feature matching loss based on the reference object feature information in the local generated description and the local reference description.

[0152] The reference object feature information includes but is limited to keywords, environment, style, atmosphere, and image quality, etc., which can be flexibly selected according to needs. The reference object features in the description generated by the feature matching loss constraint model are consistent with the local reference description. The feature matching loss can be expressed by the following formula:

[0153]

[0154] Among them, tag i is the reference object feature information in the local reference description, is a local generated description, such as the probability of a label (such as style, environment, etc.) predicted by the model, and M is the total number of labels. It should be understood that the reference object feature information may include a variety of feature labels, which can be determined according to actual needs, so as to ensure that the generated feature labels match the reference feature labels by constructing feature matching losses.

[0155] Step S750: Combining the text generation loss, the structured format loss and the feature matching loss, the parameters in the target model are adjusted to obtain an image description information extraction model.

[0156] By combining text generation loss, structured format loss and feature matching loss to construct a comprehensive loss function, the comprehensive loss function can be expressed by the following formula:

[0157] L=αL text +βL structure +γL tags Formula (7)

[0158] Among them, α, β and γ are loss weight coefficients, which can be adjusted through experiments to balance the impact of each part of the loss; L is the comprehensive loss function, L structure It is a structured format loss.

[0159] The comprehensive loss function is used to adjust the parameters in the target model until the stopping condition of the model training is met to obtain the image description information extraction model. For example, a gradient descent-based optimization method (such as the Adam optimizer) can be used to gradually adjust the parameters of the model to improve the accuracy and consistency of the generated description.

[0160] In some optional embodiments, the performance of the model can also be evaluated using a validation set to ensure that the model accurately generates image description information on unknown data. Among them, suitable evaluation indicators such as BLEU (Bilingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), CIDEr (Consensus-based Image Description Evaluation), etc. can be selected to evaluate the quality of the image description information, and the accuracy of the model for global positioning information, local positioning information, global area range, and image depth information can be evaluated at the same time.

[0161] The exemplary embodiment of the present disclosure adjusts the parameters in the target model by combining text generation loss, structured format loss and feature matching loss to ensure that the obtained image description information extraction model generates the correct image text description while making the image text description conform to the structured format.

[0162] In an exemplary embodiment, a method for determining global positioning information and local positioning information of a reference object is provided, and the process may include:

[0163] Firstly, the sample image is divided into reference image blocks, and a plurality of candidate global positioning information is determined according to the reference image blocks; secondly, the reference image blocks are divided into image sub-blocks, and a plurality of candidate local positioning information is determined according to the positions of the image sub-blocks in the reference image blocks; finally, the global positioning information corresponding to the reference object is obtained from the plurality of candidate global positioning information, and the local positioning information corresponding to the reference object is obtained from the plurality of candidate local positioning information.

[0164] The global positioning information is used to indicate the position of the reference object in the global sample image, and the local positioning information is used to indicate the position of the reference object in the image sub-block.

[0165] As an example, Fig. 9 A schematic diagram of determining the global positioning information and local positioning information of a reference object is shown below. Fig. 9 , taking the method of dividing the nine-square grid as an example to illustrate.

[0166] First, the sample image is divided into nine squares, and multiple candidate global positioning information is determined according to the position corresponding to each square of the nine squares, such as "center", "left", "right", "top", "bottom", "upper left corner", "upper right corner", "lower left corner", and "lower right corner".

[0167] Secondly, each grid is further divided into small nine-grids, and multiple candidate local positioning information is determined according to the position corresponding to each small grid of the small nine-grid, for example, "no offset", "slightly to the left", "slightly to the right", "slightly up", "slightly down", "slightly to the upper left", "slightly to the upper right", "slightly to the lower left", "slightly to the lower right".

[0168] Finally, the global positioning information corresponding to the reference object is obtained from the candidate global positioning information corresponding to the multiple nine-square grids, and the local positioning information corresponding to the reference object is obtained from the candidate local positioning information corresponding to the multiple small nine-square grids, such as Fig. 9 Examples of global positioning information and local positioning information for two reference objects are shown.

[0169] Based on the aforementioned exemplary embodiments, determining the global area range of the reference object may include:

[0170] First, the target center is determined according to the global positioning information and the local positioning information; secondly, the target area occupied by the reference object in the sample image is obtained; the target area at least includes the area size and the area type; with the target center as the center point, the global area range of the reference object is determined according to the target center and the target area.

[0171] The global region range is used to characterize the region size and region type occupied by the reference object, and the target region may include but is not limited to "small square region", "small vertical rectangular region", "small horizontal rectangular region", "medium square region", "medium vertical rectangular region", "medium horizontal rectangular region", "large square region", "large vertical rectangular region", and "large horizontal rectangular region". For each target region, the region size is preset to indicate the proportion range of the reference object corresponding to the target region that occupies the sample image.

[0172] Specifically, the area range is a relative quantity, that is, it is determined based on the size of the corresponding area of ​​the reference object in the sample image. The target center can be determined based on the center point of the global positioning information and the center point of the local positioning information. Fig. 9 As shown, the center points of the large nine-square grid and the small nine-square grid determine the target center of the global area range. After determining the target center, the target area occupied by the reference object in the sample image is obtained, and the global area range of the reference object is determined according to the target area with the target center as the center.

[0173] It should be noted that the above-mentioned method of dividing the nine-square grid is only an example, and the setting method of the specific area can be adjusted according to the actual situation, and there is no special restriction on this. At the same time, it should be understood that during the model training process, the global positioning information, local positioning information and global area range are calculated based on the specific values ​​of the corresponding items of the reference object output by the model.

[0174] The exemplary embodiments of the present disclosure are not limited to the aspect ratio or resolution of the sample image. By dividing the sample image into nine-square grids and dividing each nine-square grid into nine-square grids again, the global positioning information, local positioning information and global area range of the reference object can be determined to ensure the accuracy and consistency of the description. When the model is applied, the model can achieve a refined description of the target image in this way.

[0175] In an exemplary embodiment, the image depth information is used to sort different reference objects into a hierarchy from background to foreground, and to determine the occlusion relationship between the corresponding regions of different reference objects, so that the image description information extraction model obtained by training the model can accurately extract the image depth information. The exemplary embodiments of the present disclosure may not specifically limit the method of determining the image depth information.

[0176] In an exemplary embodiment, the method may further include: responding to the description adjustment operation, and adjusting the description text in the structured description text information according to the description adjustment operation.

[0177] According to the obtained structured global description information and local description information, structured description text information is generated. The fields in the structured description text information have clear definitions and formats, that is, they follow certain standards and specifications, making the description text information clearer and more orderly, easy to read and understand, and avoiding ambiguity. The structured description text information can also be quickly located to a specific field position, which is convenient for adjusting the description text. Therefore, the structured description text information allows the user to intuitively modify the description text according to specific needs. For example, adjust the features of people or objects in the target image, such as changing "girl" to "boy". This process adjusts the structured description text, simplifies the steps of the image editing and generation process, quickly adjusts the image content, and improves the flexibility and diversity of image generation.

[0178] As an example, take the target image as Fig.10 As shown, the instruction description information is "output the image description information of the image". Through the processing of the above exemplary embodiment, the obtained Fig.11 The structured description text information shown can also respond to the description adjustment operation of "a beautiful woman running with determination". According to the description adjustment operation, the description text "beautiful woman" in the structured description text information is directly adjusted to "boy", thereby realizing fast and intuitive adjustment of specific elements of the generated image.

[0179] After obtaining the structured description text information of the target image, the structured description text information can be used as input in the image generation scenario to guide the image generation process.

[0180] For example, in the AI ​​drawing scenario of generating graphs from graphs, the stable diffusion model is used as an AI drawing model. The target image is described by the method of the exemplary embodiment of the present disclosure, that is, the content information in the image is converted into structured description text information (refer to Fig.11 ), and then input the structured description information into the stable diffusion model, so that the stable diffusion model performs the image generation process under the guidance of the structured description information. Correspondingly, if the result of the generated image needs to be adjusted, the structured description text information can also be adjusted in response, and the adjustment result can be input into the stable diffusion model to guide the image generation process of the stable diffusion model. For example, by adjusting the structured description information, the generated image is changed from "a beautiful woman running with determination" to "a boy running with determination", thereby obtaining an image generation result that meets the user's expectations.

[0181] By generating high-quality structured description text information or modifying the structured description text information according to actual needs, the steps in the image editing and generation process are simplified, the image content is quickly adjusted, and the flexibility of image generation is improved.

[0182] The method for generating image description information in the exemplary embodiment of the present disclosure, on the one hand, acquires a target image, extracts global image features of the target image, generates global description information based on the global image features, extracts local image features of the target object in the target image, and generates local description information based on the local image features. This process generates local description information for the target object in the target image while generating global description information for the target image as a whole. The relative positions between the target objects can be clearly defined through the position information in the local description information, thereby achieving a refined description of the target image and providing high-quality information for guiding the image generation model. On the other hand, the global description information and the local description information are separated to make the description information clear and organized, and structured description text information is generated based on the global description information and the local description information, thereby increasing the clarity and readability of the description information, making it easy to modify, and improving the flexibility of generating image description information.

[0183] In an exemplary embodiment of the present disclosure, a device for generating image description information is also provided. Fig.12 As shown, the image description information generating device 1200 may include an image input module 1210, a first image processing module 1220, a second image processing module 1230 and a third image processing module 1240. Specifically:

[0184] An image input module 1210 is used to acquire a target image; a first image processing module 1220 is used to extract global image features of the target image and generate global description information based on the global image features; a second image processing module 1230 is used to extract local image features of a target object in the target image and generate local description information based on the local image features, wherein the target object is determined based on an object detection result of the target image, and the local description information includes at least object feature information and position information of the target object; and a third image processing module 1240 is used to generate structured description text information based on the global description information and the local description information.

[0185] In an exemplary embodiment of the present disclosure, the image input module 1210 is further configured to perform: receiving instruction description information of the target image, and extracting text features of the instruction description information;

[0186] The first image processing module 1220 is configured to perform: determining the global image feature according to the image fusion feature extracted from the target image; or determining the global image feature by combining the image fusion feature and the text feature.

[0187] In an exemplary embodiment of the present disclosure, if the global image feature is determined based on the image fusion feature extracted from the target image, the first image processing module 1220 is configured to execute: dividing the target image into image blocks, and performing feature encoding on each of the image blocks to obtain image block features; determining initial image features based on the features of each image block; performing denoising processing on the initial image features several times through a diffusion model, and fusing the intermediate image features corresponding to each denoising processing to obtain the image fusion feature; and determining the global image feature based on the image fusion feature.

[0188] In an exemplary embodiment of the present disclosure, if the image fusion feature and the text feature are combined to determine the global image feature, the first image processing module 1220 is configured to execute: aligning the image fusion feature and the text feature; based on the attention mechanism, according to the feature correlation between the image fusion feature and the text feature after feature alignment, fusing the image fusion feature and the text feature after feature alignment to obtain the global image feature.

[0189] In an exemplary embodiment of the present disclosure, the second image processing module 1230 is further configured to execute: performing object detection on the target image to determine the initial object in the target image; extracting a keyword vector from the instruction description information; determining a reference object corresponding to the keyword vector from the initial object based on the correlation between the global image features and the keyword vector; obtaining a correlation score between the reference object and the keyword vector, and determining the target object from the reference object based on the correlation score.

[0190] In an exemplary embodiment of the present disclosure, an image description information extraction model is used to generate global description information and local description information of the target image; it also includes a model training model, wherein the process of obtaining the image description information extraction model includes: obtaining training samples, the training samples include sample images, global reference descriptions and local reference descriptions, the local reference descriptions include at least reference object feature information and reference position information of the reference object; using the training samples to train the target model to obtain the image description information extraction model.

[0191] In an exemplary embodiment of the present disclosure, the method of training a target model using the training samples to obtain the image description information extraction model includes: inputting the sample image into the image description information extraction model to obtain a global generated description and a local generated description; constructing a text generation loss based on the global generated description and the global reference description; constructing a structured format loss based on the reference position information in the local generated description and the local reference description; the reference position information includes at least global positioning information, local positioning information, global area range and image depth information of the reference object; constructing a feature matching loss based on the reference object feature information in the local generated description and the local reference description; and adjusting the parameters in the target model in combination with the text generation loss, the structured format loss and the feature matching loss to obtain the image description information extraction model.

[0192] In an exemplary embodiment of the present disclosure, constructing a structured format loss based on the reference position information in the local generated description and the local reference description includes: determining a position loss based on the predicted global positioning in the local generated description and the global positioning information; determining an offset loss based on the predicted local positioning in the local generated description and the local positioning information; constructing a regional range loss based on the predicted global regional range in the local generated description and the global regional range; constructing a depth loss based on the predicted depth value in the local generated description and the image depth information; and determining the structured format loss by combining the position loss, the offset loss, the regional range loss and the depth loss.

[0193] In an exemplary embodiment of the present disclosure, determining the global positioning information and the local positioning information of the reference object includes: dividing the sample image into reference image blocks, and determining a plurality of candidate global positioning information according to the reference image blocks; dividing the reference image blocks into image sub-blocks, and determining a plurality of candidate local positioning information according to the positions of the image sub-blocks in the reference image blocks; acquiring the global positioning information corresponding to the reference object from the plurality of candidate global positioning information, and acquiring the local positioning information corresponding to the reference object from the plurality of candidate local positioning information.

[0194] In an exemplary embodiment of the present disclosure, determining the global area range of the reference object includes: determining a target center based on the global positioning information and the local positioning information; obtaining a target area occupied by the reference object in the sample image; the target area includes at least an area size and an area type; taking the target center as the center point, determining the global area range of the reference object based on the target center and the target area.

[0195] In an exemplary embodiment of the present disclosure, the acquiring of training samples further includes: performing image enhancement on the sample image, and performing text enhancement on the global reference description and the local reference description respectively.

[0196] In an exemplary embodiment of the present disclosure, the obtaining of training samples further includes: dividing the global reference description into multiple sub-descriptions, and dividing the local reference description into multiple sub-descriptions; wherein the length of each sub-description is less than a preset length and is semantically complete.

[0197] In an exemplary embodiment of the present disclosure, the device further includes a result adjustment module, which is used to respond to a description adjustment operation and adjust the description text in the structured description text information according to the description adjustment operation.

[0198] Since the details of the functional modules of the apparatus for generating image description information of the exemplary embodiment of the present disclosure have been described in the exemplary embodiment of the method for generating image description information, they will not be described again here.

[0199] It should be noted that although several modules or units of the image description information generation device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0200] The exemplary embodiments of the present disclosure further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned method for generating image description information is implemented.

[0201] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0202] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0203] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0204] Computer programs can be carried or transmitted through electrical, magnetic, optical, electromagnetic, infrared and other signals. Electronic devices can convert signals carrying computer programs into digital signals to run computer programs. When a computer program runs on an electronic device, its code is used to enable the electronic device to execute (more specifically, enable the processor of the electronic device to execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned method for generating image description information, which includes the following steps: acquiring a target image; extracting global image features of the target image, and generating global description information based on the global image features; extracting local image features of a target object in the target image, and generating local description information based on the local image features, wherein the target object is determined based on the object detection result of the target image, and the local description information includes at least object feature information and location information of the target object; generating structured description text information based on the global description information and the local description information.

[0205] The above method steps are executed by a computer program. On the one hand, a target image is acquired, global image features of the target image are extracted, and global description information is generated based on the global image features. Local image features of the target object in the target image are extracted, and local description information is generated based on the local image features. This process generates local description information for the target object in the target image while generating global description information for the target image as a whole. The relative positions between the target objects can be clearly defined through the position information in the local description information, so as to achieve a refined description of the target image and provide high-quality information for guiding the image generation model. On the other hand, the global description information and the local description information are separated to make the description information clear and organized, and structured description text information is generated based on the global description information and the local description information, so as to increase the clarity and readability of the description information, facilitate modification, and improve the flexibility of generating image description information.

[0206] In an exemplary embodiment of the present disclosure, the method further includes: receiving instruction description information of the target image, and extracting text features of the instruction description information; extracting global image features of the target image includes: determining the global image features based on image fusion features extracted from the target image; or, determining the global image features by combining the image fusion features and the text features.

[0207] In an exemplary embodiment of the present disclosure, if the global image feature is determined based on the image fusion feature extracted from the target image, the method includes: dividing the target image into image blocks, and performing feature encoding on each of the image blocks to obtain image block features; determining initial image features based on the features of each image block; performing denoising processing on the initial image features several times through a diffusion model, and fusing the intermediate image features corresponding to each denoising processing to obtain the image fusion feature; and determining the global image feature based on the image fusion feature.

[0208] In an exemplary embodiment of the present disclosure, if the image fusion feature and the text feature are combined to determine the global image feature, the method includes: aligning the image fusion feature and the text feature; based on the attention mechanism, according to the feature correlation between the image fusion feature and the text feature after feature alignment, fusing the image fusion feature and the text feature after feature alignment to obtain the global image feature.

[0209] In an exemplary embodiment of the present disclosure, the method also includes: performing object detection on the target image to determine the initial object in the target image; extracting a keyword vector from the instruction description information; determining a reference object corresponding to the keyword vector from the initial object based on the correlation between the global image features and the keyword vector; obtaining a correlation score between the reference object and the keyword vector, and determining the target object from the reference object based on the correlation score.

[0210] In an exemplary embodiment of the present disclosure, the global description information and local description information of the target image are generated using an image description information extraction model; wherein the process of obtaining the image description information extraction model includes: obtaining training samples, the training samples include sample images, global reference descriptions and local reference descriptions, the local reference descriptions include at least reference object feature information and reference position information of the reference object; and using the training samples to train the target model to obtain the image description information extraction model.

[0211] In an exemplary embodiment of the present disclosure, the method of training a target model using the training samples to obtain the image description information extraction model includes: inputting the sample image into the image description information extraction model to obtain a global generated description and a local generated description; constructing a text generation loss based on the global generated description and the global reference description; constructing a structured format loss based on the reference position information in the local generated description and the local reference description; the reference position information includes at least global positioning information, local positioning information, global area range and image depth information of the reference object; constructing a feature matching loss based on the reference object feature information in the local generated description and the local reference description; and adjusting the parameters in the target model in combination with the text generation loss, the structured format loss and the feature matching loss to obtain the image description information extraction model.

[0212] In an exemplary embodiment of the present disclosure, constructing a structured format loss based on the reference position information in the local generated description and the local reference description includes: determining a position loss based on the predicted global positioning in the local generated description and the global positioning information; determining an offset loss based on the predicted local positioning in the local generated description and the local positioning information; constructing a regional range loss based on the predicted global regional range in the local generated description and the global regional range; constructing a depth loss based on the predicted depth value in the local generated description and the image depth information; and determining the structured format loss by combining the position loss, the offset loss, the regional range loss and the depth loss.

[0213] In an exemplary embodiment of the present disclosure, determining the global positioning information and the local positioning information of the reference object includes: dividing the sample image into reference image blocks, and determining a plurality of candidate global positioning information according to the reference image blocks; dividing the reference image blocks into image sub-blocks, and determining a plurality of candidate local positioning information according to the positions of the image sub-blocks in the reference image blocks; acquiring the global positioning information corresponding to the reference object from the plurality of candidate global positioning information, and acquiring the local positioning information corresponding to the reference object from the plurality of candidate local positioning information.

[0214] In an exemplary embodiment of the present disclosure, determining the global area range of the reference object includes: determining a target center based on the global positioning information and the local positioning information; obtaining a target area occupied by the reference object in the sample image; the target area includes at least an area size and an area type; taking the target center as the center point, determining the global area range of the reference object based on the target center and the target area.

[0215] In an exemplary embodiment of the present disclosure, the acquiring of training samples further includes: performing image enhancement on the sample image, and performing text enhancement on the global reference description and the local reference description respectively.

[0216] In an exemplary embodiment of the present disclosure, the obtaining of training samples further includes: dividing the global reference description into multiple sub-descriptions, and dividing the local reference description into multiple sub-descriptions; wherein the length of each sub-description is less than a preset length and is semantically complete.

[0217] In an exemplary embodiment of the present disclosure, the method further includes: responding to a description adjustment operation, and adjusting the description text in the structured description text information according to the description adjustment operation.

[0218] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as a system, method, or program product. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module", or "system".

[0219] Refer to the following Fig.13 1300 according to this embodiment of the present disclosure is described. Fig.13 The electronic device 1300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0220] like Fig.13 As shown, the electronic device 1300 is in the form of a general computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310), and a display unit 1340.

[0221] The storage unit stores program codes, which can be executed by the processing unit 1310, so that the processing unit 1310 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0222] The storage unit 1320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1321 and / or a cache memory unit 1322 , and may further include a read-only memory unit (ROM) 1323 .

[0223] The storage unit 1320 may also include a program / utility 1324 having a set (at least one) of program modules 1325, such program modules 1325 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0224] Bus 1330 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0225] The electronic device 1300 may also communicate with one or more external devices 1400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1300, and / or communicate with any device that enables the electronic device 1300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1350. Furthermore, the electronic device 1300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1360. As shown, the network adapter 1360 communicates with other modules of the electronic device 1300 via a bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0226] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0227] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0228] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A method for generating image description information, characterized in that: include: Get the target image; Extracting global image features of the target image, and generating global description information according to the global image features; Extracting local image features of a target object in the target image, and generating local description information according to the local image features, wherein the target object is determined based on an object detection result of the target image, and the local description information at least includes object feature information and position information of the target object; Structured description text information is generated according to the global description information and the local description information.

2. The method according to claim 1, characterized in that The method further comprises: Receiving instruction description information of the target image, and extracting text features of the instruction description information; The extracting of global image features of the target image comprises: Determining the global image feature according to the image fusion feature extracted from the target image; Alternatively, the global image feature is determined by combining the image fusion feature and the text feature.

3. The method according to claim 2, characterized in that If the global image feature is determined based on the image fusion feature extracted from the target image, the method comprises: Dividing the target image into image blocks, and performing feature encoding on each of the image blocks to obtain image block features; Determine initial image features according to the features of each image block; Performing several noise reduction processes on the initial image features through a diffusion model, and fusing the intermediate image features corresponding to each noise reduction process to obtain the image fusion feature; The global image feature is determined based on the image fusion feature.

4. The method according to claim 2, characterized in that: If the image fusion feature and the text feature are combined to determine the global image feature, the method includes: Performing feature alignment on the image fusion feature and the text feature; Based on the attention mechanism, according to the feature correlation between the image fusion features and the text features after feature alignment, the image fusion features and the text features after feature alignment are fused to obtain the global image features.

5. The method according to claim 2, characterized in that: The method further comprises: Performing object detection on the target image to determine an initial object in the target image; Extracting a keyword vector from the instruction description information; Determining a reference object corresponding to the keyword vector from the initial objects according to the correlation between the global image feature and the keyword vector; A relevance score between the reference object and the keyword vector is obtained, and based on the relevance score, the target object is determined from the reference objects.

6. The method according to any one of claims 1 to 5, characterized in that Generate global description information and local description information of the target image using an image description information extraction model; The process of obtaining the image description information extraction model includes: Acquire a training sample, wherein the training sample includes a sample image, a global reference description, and a local reference description, wherein the local reference description includes at least reference object feature information and reference position information of a reference object; The target model is trained using the training samples to obtain the image description information extraction model.

7. The method according to claim 6, characterized in that The step of training the target model using the training samples to obtain the image description information extraction model includes: Inputting the sample image into the image description information extraction model to obtain a global generated description and a local generated description; Constructing a text generation loss according to the global generation description and the global reference description; Constructing a structured format loss according to the reference position information in the local generated description and the local reference description; the reference position information at least includes global positioning information, local positioning information, global area range and image depth information of the reference object; constructing a feature matching loss according to the reference object feature information in the local generated description and the local reference description; The text generation loss, the structured format loss and the feature matching loss are combined to adjust the parameters in the target model to obtain the image description information extraction model.

8. The method according to claim 7, characterized in that The constructing a structured format loss according to the reference position information in the local generation description and the local reference description includes: Determining a position loss based on the predicted global position in the locally generated description and the global position information; Determining an offset loss based on the predicted local location in the local generated description and the local location information; constructing a region range loss based on the predicted global region range in the local generated description and the global region range; constructing a depth loss based on the predicted depth value in the local generated description and the image depth information; The structured format loss is determined by combining the position loss, the offset loss, the area range loss and the depth loss.

9. The method according to claim 7, characterized in that: Determining the global positioning information and the local positioning information of the reference object includes: Dividing the sample image into reference image blocks, and determining a plurality of candidate global positioning information according to the reference image blocks; Divide the reference image block into image sub-blocks, and determine a plurality of candidate local positioning information according to positions of the image sub-blocks in the reference image block; The global positioning information corresponding to the reference object is acquired from the multiple candidates for global positioning information, and the local positioning information corresponding to the reference object is acquired from the multiple candidates for local positioning information.

10. The method according to claim 7, characterized in that Determining the global area scope of the reference object includes: Determining a target center according to the global positioning information and the local positioning information; Acquire a target area occupied by the reference object in the sample image; the target area at least includes an area size and an area type; Taking the target center as the center point, the global area range of the reference object is determined according to the target center and the target area.

11. The method according to claim 6, characterized in that: The obtaining of training samples further includes: Image enhancement is performed on the sample image, and text enhancement is performed on the global reference description and the local reference description respectively.

12. The method according to claim 6, characterized in that The obtaining of training samples further includes: Dividing the global reference description into a plurality of sub-descriptions, and dividing the local reference description into a plurality of sub-descriptions; The length of each sub-description is less than a preset length and is semantically complete.

13. The method according to claim 1, characterized in that The method further comprises: In response to the description adjustment operation, the description text in the structured description text information is adjusted according to the description adjustment operation.

14. A device for generating image description information, characterized in that: The device comprises: Image input module, used to obtain target image; A first image processing module, used for extracting global image features of the target image and generating global description information according to the global image features; a second image processing module, configured to extract local image features of a target object in the target image, and generate local description information according to the local image features, wherein the target object is determined based on an object detection result of the target image, and the local description information at least includes object feature information and position information of the target object; The third image processing module is used to generate structured description text information according to the global description information and the local description information.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 13 by executing the executable instructions.

Images