Image description method, device, electronic device and storage medium

Abstract

The present application provides an image description method, device, electronic device and storage medium, and relates to the field of image processing technology. The method includes: dividing an image into a foreground area and a background area, performing overlay fusion and weight assignment based on the background area to obtain an integrated background area and background weight, generating a background feature vector based on the background weight and the integrated background area, and selecting an object image corresponding to an object in the image based on the foreground area, where the object is a photographed object in the image that meets the shooting theme. Based on the background feature vector and the object-image area ratio of the area of ​​the object image to the area of ​​the image, an image description text is generated to describe the image. The present application enhances the logic of describing the spatial relationship between the object and the background through the background feature vector and the object-image area ratio to improve the accuracy of the image description.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically to an image description method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, people's demand for image recognition has increased, and the problem of low efficiency of manual image description methods has become prominent, and the application of automatic image recognition technology has emerged. In the existing technology, the description of the image is obtained by using the characteristics of the image, but there are problems such as poor logic of the image description and difficulty in the image description being close to the real intention of humans, that is, there is a problem of low accuracy of image description. Summary of the invention

[0003] The embodiments of the present application provide an image description method, device, electronic device and storage medium to solve the problem of low accuracy of image description in current methods.

[0004] An embodiment of the present application provides an image description method, the method comprising:

[0005] Divide the image into foreground area and background area;

[0006] Performing overlay fusion and weight assignment based on the background area to obtain an integrated background area and background weight;

[0007] Generate a background feature vector based on the background weight and the integrated background area;

[0008] Selecting an object image corresponding to an object in the image based on the foreground area, the object being a photographic subject in the image that meets a photographic theme;

[0009] Based on the background feature vector and the object-image area ratio of the area of ​​the object image to the area of ​​the image, an image description text is generated to describe the image.

[0010] In the above implementation process, the background area is integrated and weighted to distinguish the background area and the transition area between the background and the object, and the object-image area ratio is calculated and the object-image area ratio is used as a parameter to highlight the proportion of the object in the entire image, a background feature vector is generated based on the background weight and the integrated background area, and a description text is generated using the background feature vector and the object-image area ratio as parameters. The image can be divided into different levels by identifying the background, foreground and objects in the image, and the parameters and weights used for the description text are determined according to the relationship between the different levels, thereby improving the logic of describing the spatial relationship between the object and the background in the image, so as to improve the accuracy of the image description.

[0011] Optionally, dividing the image into a foreground area and a background area includes:

[0012] Setting a predetermined number of anchor points on the image;

[0013] Generating each candidate region based on the predetermined number of anchor points;

[0014] Based on the photographed subject, each candidate area is divided into a foreground area and a background area using a pre-trained model.

[0015] In the above implementation process, the predetermined number of anchor points and the pre-trained model are used to divide the candidate areas into foreground areas and background areas that match the shooting subject. The image is divided into different areas based on the shooting subject, and different processing methods are used for different areas, which can improve the accuracy of image description.

[0016] Optionally, performing coverage fusion and weight assignment based on the background area to obtain an integrated background area and background weight includes:

[0017] Taking a union of each group of background regions that overlap in all background regions to obtain at least one first region;

[0018] Obtaining an integrated background area based on the first area and second areas dispersed in the background area;

[0019] A weight is assigned based on the integrated background area to obtain a background weight of the integrated background area.

[0020] In the above implementation process, taking the union of each group of overlapping background areas can avoid repeated calculation of the overlapping background areas, and assigning weights after integrating the integrated background areas can distinguish the transition areas and other parts in the background areas, thereby improving the accuracy of image description based on the background weights.

[0021] Optionally, selecting an object image corresponding to the object in the image based on the foreground area includes:

[0022] Extracting image features of the foreground area based on a convolutional layer in a training network;

[0023] Using the image features as input to a fully connected layer in the training network to obtain classified image features;

[0024] The object image is selected based on the classified image features by a non-maximum suppression method.

[0025] In the above implementation process, by searching for the local maximum of the image features in the foreground area based on the image features of the foreground area, the maximum value of the image features in the foreground area is suppressed, and the non-maximum suppression method is used to detect and identify multiple objects in the foreground area, thereby improving the efficiency of selecting object images.

[0026] Optionally, generating an image description text based on the background feature vector and the object-image area ratio to describe the image includes:

[0027] After the background feature vector is input into the long short-term memory network, the object-image area ratio corresponding to the background feature vector is operated with a forget gate in the long short-term memory network until the image description text is generated.

[0028] In the above implementation process, the long short-term memory network is used as the training network. The gradient explosion and gradient disappearance are reduced through the forget gate and update gate in the long short-term memory network, so that the state of the image within a relatively long period of time can be obtained, thereby improving the accuracy of image description.

[0029] Optionally, before dividing the image into a foreground area and a background area, the method further comprises:

[0030] Data cleaning is performed on the image description text to obtain a cleaned description text to verify the image description text.

[0031] In the above implementation process, the image description text is cleaned to delete duplicate information and correct errors, and the image description is verified to improve the accuracy of the image description.

[0032] An embodiment of the present application provides an image description device, the image description device comprising:

[0033] A segmentation module, used for dividing the image into a foreground area and a background area;

[0034] A fusion assignment module, used for performing covering fusion and weight assignment based on the background area to obtain an integrated background area and background weight;

[0035] A background processing module, used for generating a background feature vector based on the background weight and the integrated background area;

[0036] A foreground processing module, used to select an object image corresponding to an object in the image based on the foreground area, wherein the object is a photographic subject in the image that meets the photographic theme;

[0037] The description generation module is used to generate image description text to describe the image based on the background feature vector and the object-image area ratio of the area of ​​the object image to the area of ​​the image.

[0038] In the above implementation process, by integrating and assigning weights to the background area to distinguish the background area and the transition area between the background and the object, and calculating the object-image area ratio and using the object-image area ratio as a parameter, the proportion of the object in the entire image can be highlighted, a background feature vector is generated based on the background weight and the integrated background area, and a description text is generated using the background feature vector and the object-image area ratio as parameters. The image can be divided into different levels by identifying the background, foreground and objects in the image, and the parameters and weights used for the description text are determined according to the relationship between the different levels, thereby improving the logic of describing the spatial relationship between the object and the background in the image, so as to improve the accuracy of the image description.

[0039] Optionally, the partitioning module is used to:

[0040] Setting a predetermined number of anchor points on the image;

[0041] Generate each candidate region based on the anchor point;

[0042] Based on the photographed subject, each candidate area is divided into a foreground area and a background area using a pre-trained model.

[0043] In the above implementation process, the predetermined number of anchor points and the pre-trained model are used to divide the candidate areas into foreground areas and background areas that match the shooting subject. The image is divided into different areas based on the shooting subject, and different processing methods are used for different areas, which can improve the accuracy of image description.

[0044] Optionally, the fusion assignment module is used to:

[0045] Taking a union of each group of background regions that overlap in all background regions to obtain at least one first region;

[0046] Obtaining an integrated background area based on the first area and second areas dispersed in the background area;

[0047] A weight is assigned based on the integrated background area to obtain a background weight of the integrated background area.

[0048] In the above implementation process, taking the union of each group of overlapping background areas can avoid repeated calculation of the overlapping background areas, and assigning weights after integrating the integrated background areas can distinguish the transition areas and other parts in the background areas, thereby improving the accuracy of image description based on the background weights.

[0049] Optionally, the foreground processing module is used to:

[0050] Extracting image features of the foreground area based on a convolutional layer in a training network;

[0051] Using the image features as input to a fully connected layer in the training network to obtain classified image features;

[0052] The object image is selected based on the classified image features by a non-maximum suppression method.

[0053] In the above implementation process, by searching for the local maximum of the image features in the foreground area based on the image features of the foreground area, the maximum value of the image features in the foreground area is suppressed, and the non-maximum suppression method is used to detect and identify multiple objects in the foreground area, thereby improving the efficiency of selecting object images.

[0054] Optionally, the description generating module is specifically used for:

[0055] After the background feature vector is input into the long short-term memory network, the object-image area ratio corresponding to the background feature vector is operated with the forget gate and the update gate in the long short-term memory network until the image description text is generated.

[0056] In the above implementation process, the long short-term memory network is used as the training network, and the gradient explosion and gradient disappearance can be reduced through the forgetting gate in the long short-term memory network. The long short-term memory network can determine the retention or forgetting of the image description text, and can obtain the state of the image within a long period of time, thereby improving the accuracy of the image description.

[0057] Optionally, the image description device further includes:

[0058] The cleaning module is used to perform data cleaning on the image description text to obtain a cleaned description text so as to verify the image description text.

[0059] In the above implementation process, the image description text is cleaned to delete duplicate information and correct errors, and the image description is verified to improve the accuracy of the image description.

[0060] This embodiment further provides an electronic device, which includes a memory and a processor. The memory stores program instructions, and when the processor runs the program instructions, the steps in any of the above methods are executed.

[0061] This embodiment further provides a storage medium, in which computer program instructions are stored. When the computer program instructions are executed by a processor, the steps in any of the above methods are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0063] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0064] Figure 1 A flowchart of an image description method provided in an embodiment of the present application.

[0065] Figure 2 A flowchart of an image segmentation step provided in an embodiment of the present application.

[0066] Figure 3 A flowchart of the steps for processing the background area provided in an embodiment of the present application.

[0067] Figure 4 A flowchart of the steps for processing the foreground area provided in an embodiment of the present application.

[0068] Figure 5 A schematic diagram of an image description device provided in an embodiment of the present application.

[0069] Legend: 70-image description device; 701-division module; 702-fusion assignment module; 703-background processing module; 704-foreground processing module; 705-description generation module; 706-cleaning module. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0071] In the description of the present application, it should be noted that the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0072] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by implementing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description and the drawings.

[0073] Image description technology is widely used in the fields of early childhood education, blind assistance, etc. It can also provide technical support for video description technology and video question-answering technology. In the process of image description, in order to improve the accuracy of image description, that is, to make the image description closer to the real intention of human beings, the embodiment of this application provides a method for image description, please refer to Figure 1 , Figure 1 A flowchart of an image description method provided in an embodiment of the present application, the image description method comprises the following sub-steps:

[0074] Step S1: Divide the image into a foreground area and a background area.

[0075] In step S1, the area in the image that matches the subject is defined as the foreground area, and the remaining part outside the subject is defined as the background area. For example, in an image that includes people and mountains, if the subject is a portrait, then the foreground area corresponds to the area corresponding to the person in the image, and the background area is the other area outside the person. It can be understood that the subject can be changed subjectively and used as an input for dividing the foreground area or the background area. The same image can have different shooting themes. Under different shooting themes, the foreground area and the background area in the same image will change.

[0076] Take an image that includes people and mountains as an example. If the subject is the mountain, then the foreground area corresponds to the area corresponding to the mountain in the image, and the background area is other areas outside the mountain.

[0077] It is understandable that, when selecting an image, in order to improve the spatial relationship between the object and the background in the foreground area, the area occupied by the object in the entire image should not be too large.

[0078] Please see Figure 2 , Figure 2 A flowchart of an image segmentation step provided in an embodiment of the present application. Optionally, step S1 includes the following sub-steps:

[0079] Step S11: Setting a predetermined number of anchor points on the image.

[0080] Step S12: Generate each candidate region based on a predetermined number of anchor points.

[0081] In step S11 and step S12, anchor points can be selected on the image according to the subject of the shooting, and different candidate areas can be formed by connecting the anchor points. As an implementation method, pixel points can be selected in sequence from any corner (e.g., the upper left corner) of the image at intervals of a predetermined distance, and the pixel points can be used as anchor points. The predetermined distance can be set according to the size of the image and the complexity of the contour of the object in the image. With the anchor point as the center, an area of ​​a predetermined geometric shape around the anchor point is selected as a candidate area. The predetermined geometric shape can be any geometric shape such as a rectangle or a circle.

[0082] It is understandable that the more anchor points there are, the finer the candidate areas are, and the closer the foreground area obtained by dividing based on the candidate areas is to the subject of the photograph, that is, the more accurate the division of the foreground area and the background area is. The size of the predetermined number can be set according to the size of the object corresponding to the subject of the photograph and the complexity of the contour. The larger the size of the object and the more complex the contour, the larger the predetermined number. For example, the predetermined number can be any number from 6 to 12.

[0083] Step S13: Based on the photographed subject, each candidate area is divided into a foreground area and a background area using a pre-trained model.

[0084] In step S13, the candidate regions formed based on a predetermined number of anchor points are used as inputs of the pre-training model, and each candidate region outputs a division result of the candidate region after passing through the pre-training model.

[0085] It can be understood that the pre-trained model can adopt any convolutional neural network as needed. For example, in this embodiment, the open source model Faster-R-CNN can be used, and the ROI pooling (Region of interest pooling) operation based on Faster-R-CNN outputs whether each candidate area belongs to the foreground area or the background area.

[0086] Faster-R-CNN can effectively locate each candidate region and obtain image features according to feature indexing of each candidate region, which can reduce the time consumption of convolution calculation, that is, improve computing efficiency.

[0087] Step S2: Perform overlay fusion and weight assignment based on the background area to obtain the integrated background area and background weight.

[0088] Please see Figure 3 , Figure 3A flowchart of a background area processing step provided in an embodiment of the present application. Optionally, step S2 includes the following sub-steps:

[0089] Step S21: taking a union of each group of overlapping background regions among all background regions to obtain at least one first region.

[0090] In step S2, there may be overlap in the background area. When assigning weights, if weights are assigned to all parts of the background area, there will be repeated assignments. Therefore, in order to improve the accuracy of weight assignment to the background area, the background area is covered and fused and then weights are assigned.

[0091] Overlapping means that at least two background regions partially overlap, and the overlapping background regions can be regarded as a group. Taking the union of all the overlapping background regions in each group can avoid repeated weight assignment for the overlapping parts, and improve the accuracy of weight assignment for the background regions.

[0092] Step S22: obtaining an integrated background region based on the first region and the second regions dispersed in the background region.

[0093] It can be understood that there are scattered parts in the background area. The scattered second area is the background area without overlap among all the background areas. The collection of the first area and the second area is the integrated background area.

[0094] Step S23: performing weight assignment based on the integrated background area to obtain a background weight of the integrated background area.

[0095] In step S23, when assigning weights to the integrated background area, in order to highlight the background area, the first area is assigned a first weight value, the weight of the part where the object and the background intersect on the image or the object transitions to the background is assigned a second weight value, and the weight of other parts (including the second area) is assigned a third weight value, wherein the first weight value is greater than the second weight value, the first weight value is greater than the third weight value, and the sizes of the third weight value and the second weight value can be set arbitrarily. Under this weight assignment method, the background area is highlighted.

[0096] Step S3: Generate a background feature vector based on the background weight and the integrated background area.

[0097] In step S3, any convolutional neural network can be used as a training model, and the background weight and the integrated background area are used as inputs of the training model to obtain the background feature vector of the integrated background area of ​​the training model.

[0098] It can be understood that in step S3, when the background area is input into the training model, since the size of the convolution calculation is fixed, in order to facilitate the convolution calculation, the size of all input background areas needs to be set to a uniform size.

[0099] The training network may adopt any convolutional neural network. As an implementation method, the training network may adopt the open source model Faster-R-CNN. In step S3, the image features of the foreground area are extracted in the convolution layer of Faster-R-CNN. Faster-R-CNN can effectively locate the background area and obtain the image features of the background area according to the feature index of the foreground area, which can reduce the time consumption of convolution calculation, that is, improve the operation efficiency.

[0100] Please continue to see Figure 1 , step S4: selecting an object image corresponding to the object in the image based on the foreground area, the object being a photographic subject in the image that meets the photographic theme.

[0101] Please see Figure 4 , Figure 4 A flowchart of a foreground area processing step provided in an embodiment of the present application. Step S4 includes the following sub-steps:

[0102] Step S41: extracting image features of the foreground area based on the convolutional layer in the training network.

[0103] In step S41, the training network can use any convolutional neural network. As an implementation method, the training network in this embodiment can use the open source model Faster-R-CNN. In step S41, the image features of the foreground area are extracted in the convolution layer of Faster-R-CNN. As described in step S13, Faster-R-CNN can effectively locate the foreground area, and obtain the image features of the foreground area according to the feature index of the foreground area, which can reduce the time consumption of convolution calculation, that is, improve the operation efficiency.

[0104] It can be understood that in step S41, when the background area is input into the training network, since the size of the convolution calculation is fixed, in order to facilitate the convolution calculation, the size of all input foreground areas needs to be set to a uniform size.

[0105] Step S42: Using the image features as input to the fully connected layer in the training network to obtain classified image features.

[0106] It is understandable that the purpose of using the image features of the foreground area as the input of the fully connected layer to classify the image features is to allow the training network to self-learn on the classification task and thus have the ability to extract features.

[0107] Step S43: Selecting the object image by non-maximum suppression method based on the classified image features.

[0108] In step S43, the principle of the non-maximum suppression method is to search for the local maximum of the image features in the foreground area based on the image features of the foreground area, and suppress the maximum of the image features in the foreground area. The non-maximum suppression method can detect and identify multiple objects in the foreground area, improve the efficiency of selecting object images, and avoid wasting computing resources when calculating the object-image area ratio based on the foreground area selected after removing the maximum value.

[0109] It should be noted that the models used in the training process in step S3 and step S41 can be the same, but the training processes in step S3 and step S41 are independent and do not interfere with each other.

[0110] Step S5: Based on the background feature vector and the object-image area ratio of the area of ​​the object image to the area of ​​the image, generate an image description text to describe the image.

[0111] Optionally, step S5 includes: after the background feature vector is input into the long short-term memory network, the object-image area ratio corresponding to the background feature vector is operated with a forget gate in the long short-term memory network until the image description text is generated.

[0112] As an implementation method, after the background feature vector is input into a long short-term memory (LSTM) network, during the update gate matrix operation in the long short-term memory network, the result obtained by multiplying the background feature vector by the area ratio is multiplied by the variable in the update gate, and the image description text is output at the output end of the long short-term memory network.

[0113] It can be understood that by using the long short-term memory network as a training network, that is, using the long short-term memory network as a training model, the gradient explosion and gradient disappearance can be reduced through the forget gate in the long short-term memory network. The long short-term memory network can determine the retention or forgetting of the image description text to obtain the state of the image within a longer period of time, thereby improving the accuracy of the image description.

[0114] It can be understood that using the object-image area ratio as a parameter to perform operations with the forget gate can highlight objects in the foreground area.

[0115] Optionally, before step S1, the image description method further includes step S6: performing data cleaning on the image description text to obtain a cleaned description text to verify the image description text.

[0116] It is understandable that this application aims to enhance the logic of describing the spatial relationship between objects and backgrounds to improve the accuracy of image description. Therefore, it is required that the images in the training data contain backgrounds and objects, or at least two types of objects. Therefore, images that only contain one object (such as close-ups of human faces) are to be eliminated. In addition, if humans cannot recognize the background in the image, such images also need to be eliminated.

[0117] For the artificially prepared text description of the above image, when the description sentence does not include a description of the background, for example, sentences without background descriptions such as "a cat eating" and "a person" should also be deleted, and text descriptions that include both objects and backgrounds such as "a person in a bedroom" and "a cat eating on a table" should be selected or retained as much as possible. It can be understood that after the image passes through steps S1-S5, the trained image text description is obtained, and the accuracy of the image description after the trained model can be determined by comparing the artificially prepared text description of the image with the trained image text description.

[0118] It should be noted that if the above data to be eliminated are completely removed, the robustness of the model will be reduced. Therefore, the amount of data that does not meet the requirements accounts for 1%-5% of the total data volume.

[0119] Please see Figure 5 , Figure 5 A schematic diagram of an image description device provided in an embodiment of the present application. The image description device 70 may include:

[0120] The division module 701 is used to divide the image into a foreground area and a background area.

[0121] The fusion and assignment module 702 is used to perform covering fusion and weight assignment based on the background area to obtain the integrated background area and background weight.

[0122] The background processing module 703 is used to generate a background feature vector based on the background weight and the integrated background area.

[0123] The foreground processing module 704 is used to select an object image corresponding to an object in the image based on the foreground area, where the object is a photographic subject in the image that meets the photographic theme.

[0124] The description generating module 705 is used to generate image description text to describe the image based on the background feature vector and the object-image area ratio of the area of ​​the object image to the area of ​​the image.

[0125] Optionally, the division module 701 can be used to: set a predetermined number of anchor points on the image, generate various candidate areas based on the anchor points, and divide the various candidate areas into foreground areas and background areas based on the shooting subject using a pre-trained model.

[0126] Optionally, the fusion assignment module 702 may be used to:

[0127] A union is taken for each group of overlapping background areas in all background areas to obtain at least one first area, an integrated background area is obtained based on the first area and second areas scattered in the background area, and weights are assigned based on the integrated background area to obtain a background weight of the integrated background area.

[0128] Optionally, the foreground processing module 704 may be used to:

[0129] The image features of the foreground area are extracted based on the convolutional layer in the training network, and the image features are used as the input of the fully connected layer in the training network to obtain the classified image features. The object image is selected based on the classified image features through the non-maximum suppression method.

[0130] Optionally, the description generating module 705 may be specifically used for:

[0131] After the background feature vector is input into the long short-term memory network, the object-image area ratio corresponding to the background feature vector is operated with the forget gate and the update gate in the long short-term memory network until the image description text is generated.

[0132] Optionally, the image description device 70 may further include:

[0133] The cleaning module 706 is used to perform data cleaning on the image description text to obtain a cleaned description text for verifying the image description text.

[0134] This embodiment further provides a storage medium, in which computer program instructions are stored. When the computer program instructions are executed by a processor, the steps in any of the above methods are executed.

[0135] This embodiment further provides an electronic device, which includes a memory and a processor. The memory stores program instructions, and when the processor runs the program instructions, the steps in any of the above methods are executed.

[0136] In summary, the embodiments of the present application provide an image description method, device, electronic device and storage medium, which relate to the field of image processing technology. The image description method includes: dividing an image into a foreground area and a background area, performing overlay fusion and weight assignment based on the background area to obtain an integrated background area and background weight, generating a background feature vector based on the background weight and the integrated background area, and selecting an object image corresponding to an object in the image based on the foreground area, wherein the object is a photographed object in the image that meets the photographing theme. Based on the background feature vector and the object-image area ratio of the area of ​​the object image to the area of ​​the image, an image description text is generated to describe the image.

[0137] In the above implementation process, the background area is integrated and weighted to distinguish the background area and the transition area between the background and the object, and the object-image area ratio is calculated and the object-image area ratio is used as a parameter to highlight the proportion of the object in the entire image, a background feature vector is generated based on the background weight and the integrated background area, and a description text is generated using the background feature vector and the object-image area ratio as parameters. The image can be divided into different levels by identifying the background, foreground and objects in the image, and the parameters and weights used for the description text are determined according to the relationship between the different levels, thereby improving the logic of describing the spatial relationship between the object and the background in the image, so as to improve the accuracy of the image description.

[0138] In several embodiments provided in the present application, it should be understood that the disclosed device can also be implemented in other ways. The device embodiments described above are merely schematic, for example, the block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices according to the multiple embodiments of the present application. In this regard, each box in the block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram, and the combination of the block diagrams, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0140] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, this embodiment also provides a computer program instruction stored in a readable storage medium, and when the computer program instruction is read and executed by a processor, the steps in any method described in the block data storage method are executed. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0141] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image description method, characterized in that: The method comprises: Divide the target image into a foreground area and a background area; Performing overlay fusion and weight assignment based on the background area to obtain an integrated background area and background weight; Generate a background feature vector based on the background weight and the integrated background area; Selecting an object image corresponding to an object in the target image based on the foreground area, wherein the object is a photographic subject in the target image that meets the photographic theme; Based on the background feature vector and the object-image area ratio of the area of ​​the object image to the area of ​​the target image, generate an image description text to describe the target image; The method of performing coverage fusion and weight assignment based on the background area to obtain an integrated background area and background weight includes: Taking a union of each group of background regions that overlap in all background regions to obtain at least one first region; Obtaining an integrated background area based on the first area and second areas dispersed in the background area; Performing weight assignment based on the integrated background area to obtain a background weight of the integrated background area; The step of assigning weights based on the integrated background area to obtain a background weight of the integrated background area includes: Assigning a first weight value to the first area, assigning a second weight value to the weight of the part where the object intersects with the background on the image or where the object transitions to the background, and assigning a third weight value to the weight of the remaining part; wherein the first weight value is greater than the second weight value, and the first weight value is greater than the third weight value; The step of generating an image description text based on the background feature vector and the object-image area ratio to describe the target image includes: After the background feature vector is input into the long short-term memory network, the object-image area ratio corresponding to the background feature vector is operated with the forget gate and the update gate in the long short-term memory network until the image description text is generated.

2. The image description method according to claim 1, characterized in that: The step of dividing the target image into a foreground area and a background area comprises: Setting a predetermined number of anchor points on the target image; Generating each candidate region based on the predetermined number of anchor points; Based on the photographed subject, each candidate area is divided into a foreground area and a background area using a pre-trained model.

3. The image description method according to claim 1, characterized in that: The selecting, based on the foreground area, an object image corresponding to the object in the target image comprises: Extracting image features of the foreground area based on a convolutional layer in a training network; Using the image features as input to a fully connected layer in the training network to obtain classified image features; The object image is selected based on the classified image features by a non-maximum suppression method.

4. The image description method according to claim 1, characterized in that: Before dividing the target image into a foreground area and a background area, the method further includes: Data cleaning is performed on the image description text to obtain a cleaned description text to verify the image description text.

5. An image description device, characterized in that: The device comprises: A segmentation module, used for dividing the target image into a foreground area and a background area; A fusion assignment module, used for performing covering fusion and weight assignment based on the background area to obtain an integrated background area and background weight; A background processing module, used for generating a background feature vector based on the background weight and the integrated background area; A foreground processing module, used to select an object image corresponding to an object in the target image based on the foreground area, wherein the object is a photographic subject in the target image that meets the photographic theme; A description generating module, configured to generate an image description text to describe the target image based on the background feature vector and the object-image area ratio of the area of ​​the object image to the area of ​​the target image; In the process of performing covering fusion and weight assignment on the background area to obtain the integrated background area and background weight, the fusion assignment module is specifically used to: take the union of each group of overlapping background areas in all background areas to obtain at least one first area; obtain the integrated background area based on the first area and the second areas scattered in the background area; perform weight assignment based on the integrated background area to obtain the background weight of the integrated background area; In the process of assigning weights based on the integrated background area to obtain the background weight of the integrated background area, the description generation module is specifically used to: assign a first weight value to the first area, assign a second weight value to the weight of the part where the object intersects with the background on the image or the object transitions to the background, and assign a third weight value to the weight of the remaining part; wherein the first weight value is greater than the second weight value, and the first weight value is greater than the third weight value; In the process of generating an image description text based on the background feature vector and the object-image area ratio to describe the target image, the description generation module is specifically used to: after the background feature vector is input into the long short-term memory network, the object-image area ratio corresponding to the background feature vector is operated with the forget gate and the update gate in the long short-term memory network until the image description text is generated.

6. The image description device according to claim 5, characterized in that: The partitioning module is used for: Setting a predetermined number of anchor points on the target image; Generating each candidate region based on the predetermined number of anchor points; Based on the photographed subject, each candidate area is divided into a foreground area and a background area using a pre-trained model.

7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps in the image description method according to any one of claims 1 to 4 are executed.

8. A storage medium, characterized in that: The storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the steps of the image description method according to any one of claims 1 to 4 are executed.

Images