Method and device for improving picture description effect

By obtaining the object names and detection boxes in the picture, determining the importance sorting and generating intermediate text, combined with the pre-trained language model, the key problems of image description model grasping key problems and data scarcity are solved, and the image description effect and model decoding capabilities are improved.

CN114119726BActive Publication Date: 2024-11-01AISPEECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111200753.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-13
Publication Date
2024-11-01
Estimated Expiration
2041-10-13

AI Technical Summary

Technical Problem

Existing picture description models are difficult to grasp the focus of the picture, and due to scarce data, the output is not smooth or single-form sentences.

Method used

By obtaining all the objects names and detection boxes in the picture to be described, determining the importance of objects, generating intermediate text, and fusing the pre-trained language model on the decoding end of the transformer model, directly telling the key points of the model picture and reducing the difficulty of generating the model training.

Benefits of technology

It improves the effect of picture description, the generated text is more smooth and informative, reduces the difficulty of model training, and improves the quality of description.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114119726B_ABST
    Figure CN114119726B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for improving the effect of picture description, wherein a method for improving the effect of picture description includes: obtaining the names of all objects in the picture to be described and the detection frames of all the objects; determining the importance ranking of all the objects at least based on the detection frames of all the objects; and generating intermediate text based on the importance ranking and the names of all the objects. By obtaining the names of all the objects in the picture to be described and the detection frames of all the objects and determining the importance ranking of all the objects, and then generating the intermediate text, it is possible to directly tell the model the key points of the picture and improve the description effect of the picture.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of picture description, and in particular, relates to a method and device for improving picture description effect. Background Art

[0002] Image caption refers to describing an image in text language. In essence, it is a method of "translating" image information into text information, similar to kindergarten children's picture description, which simply describes the content of a picture, such as: a monkey climbing a tree. In recent years, image captioning has opened up application scenarios in image retrieval, education, medical assistance, news media, smart transportation, etc. With the development of artificial intelligence, image captioning will have more extensive applications.

[0003] At present, image description is still a very challenging problem in the field of artificial intelligence. In terms of models, the best model currently is the image description based on the transformer model. It first uses the image detection model to extract image features. The features here are vectors or matrices with fixed dimensions, which contain the names of all objects in the image as well as their size and location information. The image features are then input into the transformer model, and the target end is the text description of the image. Although this approach can retain the image information to the greatest extent possible, the model will not be able to grasp the key points when describing the image. For example, the model outputs "a clock in front of the school", while normal people are concerned about "a school by the lake".

[0004] In addition, data scarcity is an important reason that restricts the development of image description. Compared with the data in the field of natural language processing, which ranges from millions to hundreds of millions, the data sets used in image description research are generally around 100,000. The data scale is too small, resulting in the model usually outputting incoherent or monotonous sentences. For example, the model outputs the incoherent sentence "a bottle of beer sitting on the table." Summary of the invention

[0005] The embodiments of the present invention provide a method and device for improving picture description effect, so as to alleviate the above problems.

[0006] In a first aspect, an embodiment of the present invention provides a method for improving the effect of image description, comprising: obtaining the names of all objects in the image to be described and the detection frames of all the objects; determining the importance ranking of all the objects at least based on the detection frames of all the objects; and generating an intermediate text based on the importance ranking and the names of all the objects.

[0007] In a second aspect, an embodiment of the present invention provides a device for improving the effect of image description, comprising: an acquisition program module, configured to obtain the names of all objects in the image to be described and the detection frames of all the objects; a sorting program module, configured to determine the importance ranking of all the objects based at least on the detection frames of all the objects; and a generation program module, configured to generate an intermediate text based on the importance ranking and the names of all the objects.

[0008] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method for improving the picture description effect of any embodiment of the present invention.

[0009] In a fourth aspect, an embodiment of the present invention further provides a computer program product, comprising a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the steps of the method for improving the picture description effect of any embodiment of the present invention.

[0010] The method and device of the present application obtain the names of all objects in the picture to be described and the detection frames of all objects and determine the importance ranking of all objects, and then generate intermediate text, so as to directly tell the model the key points of the picture and improve the description effect of the picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flowchart of a method for improving picture description effect provided by an embodiment of the present invention;

[0013] Figure 2 A flowchart of another method for improving picture description effect provided by an embodiment of the present invention;

[0014] Figure 3 A method example diagram of a specific example of a method for improving picture description effect provided by an embodiment of the present invention;

[0015] Figure 4 A block diagram of a device for improving picture description effect provided by an embodiment of the present invention;

[0016] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Please refer to Figure 1 , which shows a flow chart of an embodiment of a method for improving the picture description effect of the present application. The method for improving the picture description effect of this embodiment can be applied to terminals with picture description functions, such as smart phones, tablets, computers, servers, etc.

[0019] like Figure 1 As shown, in step 101, the names of all objects in the picture to be described and the detection frames of all the objects are obtained;

[0020] In step 102, the importance ranking of all the objects is determined based at least on the detection frames of all the objects;

[0021] In step 103, an intermediate text is generated based on the importance ranking and the names of all the objects.

[0022] In this embodiment, for step 101, the device for improving the picture description effect obtains the names of all objects in the picture to be described and the detection frames of all objects. For example, the size of the picture to be described is 500*500. After picture detection, it is obtained that the objects in the picture to be detected include a table, a bottle and flowers, and the detection frames corresponding to the table, the bottle and the flowers. For example, the width of the picture to be detected is represented by W, and the height is represented by H. The detection frame corresponding to the object in the picture to be detected is represented by (x, y, w, h), where x and y are the coordinates of the center point of the detection frame, w is the width of the detection frame, and h is the height of the detection frame. For example, the detection frames corresponding to the table, the bottle and the flowers are [[300, 200, 400, 300], [280, 200, 100, 200], [200, 200, 300, 200]].

[0023] Afterwards, for step 102, the device for improving the image description effect determines the importance ranking of all objects based at least on the detection frames of all objects, for example, by fusing the area score, position score and attention probability of the object to determine the importance score of the object, thereby determining the importance ranking of the object.

[0024] Finally, for step 103, the device for improving the picture description effect generates an intermediate text based on the importance ranking and the names of all objects. For example, all the objects in the picture to be described are tables, bottles and flowers, among which the importance score of flowers is the first, the importance score of tables is the second, and the importance score of bottles is the third. According to the importance ranking, they are flowers, tables, and bottles. After that, the text flowers, table, and bottle are input into the picture description model.

[0025] The method of this embodiment obtains the names of all objects in the picture to be described and the detection frames of all objects, determines the importance ranking of all objects, and then generates intermediate text, so as to directly tell the model the key points of the picture and improve the description effect of the picture.

[0026] Further references Figure 2 , which shows a flowchart of another method for improving the picture description effect provided by an embodiment of the present application. Figure 1 A flowchart of the steps further defined in the process of "determining the importance ranking of all objects based at least on the detection boxes of all objects".

[0027] like Figure 2 As shown, in step 201, the area score of each object is determined based on the size of the detection frame of each object among all the objects occupied by the picture to be described;

[0028] In step 202, a position score of each object is determined based on the distance between the detection frame of each object and the center of the image to be described;

[0029] In step 203, the attention probability of each object in the preset picture description training set is obtained;

[0030] In step 204, all the objects are ranked in importance based on the area score of each object, the position score of each object, and the attention probability of each object.

[0031] In this embodiment, for step 201, the device for improving the picture description effect determines the area score of each object based on the size of the detection frame of each object among all objects in the picture to be described. For example, the larger the object in the picture to be detected, the higher the score.

[0032] Then, for step 202, the device for improving the picture description effect determines the position score of each object based on the distance between the detection frame of each object and the center of the picture to be described. For example, the closer the object is to the center of the picture to be detected, the higher the score.

[0033] Afterwards, for step 203, the device for improving the picture description effect obtains the attention probability of each object in the preset picture description training set. For example, the preset picture description training set contains a large number of pictures and corresponding text descriptions. If an object appears in a picture in the preset picture description training set and is also mentioned in the text description, it is considered that the object is paid attention to, and the attention probability of each object in the picture to be described is calculated based on the formula for calculating the attention probability.

[0034] Finally, for step 204, the device for improving the image description effect ranks all objects by importance based on the area score of each object, the position score of each object, and the attention probability of each object. For example, each object is integrated with the corresponding area score, position score, and attention probability to obtain the total importance score of each object, and all objects are ranked by importance.

[0035] The method of this embodiment ranks all objects by importance based on the area score of each object, the position score of each object, and the attention probability of each object, so as to find the focus of the picture.

[0036] In the method described in the above embodiment, obtaining the attention probability of each object in the preset picture description training set includes:

[0037] The preset picture description training set includes multiple pictures and the text description corresponding to each picture. If an object appears in a picture in the preset picture description training set and is mentioned in the text description corresponding to a picture in the preset picture description training set, then the object is determined to be paid attention to. The attention probability of the object is calculated based on the number of attentions to the object and the number of appearances of the object in the preset picture description training set. Among them, if an object appears in a picture and is mentioned in another picture, it can also be determined to be paid attention to.

[0038] It should be noted that the attention probability can actually be considered as a relatively fixed constant for the big data of the picture description training set. If the picture description training set is fixed, the attention probability of each object in the picture description training set is also fixed. The larger the amount of data in the picture description training set, the more accurate the attention probability of each object. This application will not go into details here.

[0039] In the method described in the above embodiment, after generating the intermediate text based on the importance ranking and the names of all the objects, the method further includes:

[0040] The intermediate text is input into the image description model to obtain the output image description text. For example, the intermediate text can use rules to connect the detected objects through language, such as a bouquet of flowers on the table. Through the conversion of the intermediate text, the training set of the image description is switched from the original image to text to text.

[0041] The method of this embodiment can reduce the difficulty of training the generation model by inputting the intermediate text into the image description model to obtain the output image description text.

[0042] In the method described in the above embodiment, the image description model is a transformer model, and the decoding end of the transformer model is fused with a pre-trained language model through a preset fusion method.

[0043] The method of this embodiment fuses a pre-trained language model through a preset fusion method at the decoding end of the transformer model, thereby improving the model decoding capability.

[0044] In the above embodiments, the pre-trained language model is BERT or XLNet, and the preset fusion method includes DeepFusion or ColdFusion.

[0045] In the method described in any of the above embodiments, the intermediate text includes using preset rules to connect all the objects through relational language according to the importance sorting, and the relationship includes a positional relationship, for example, there is a table around the air conditioner or there is a notebook on the table.

[0046] The method of this embodiment uses preset rules to link all the objects together through relational language according to the importance ranking, thereby improving the description effect of the picture.

[0047] It should be noted that the above method steps are not used to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the opposite order to the specified step order, and this application has no limitation on this.

[0048] The following describes some problems encountered by the inventor in the process of implementing the present invention and a specific embodiment of the solution finally determined, so that those skilled in the art can better understand the solution of the present application.

[0049] In the process of implementing this application, the inventor found that the defects in the prior art are mainly caused by the following reasons:

[0050] Data scarcity is an important reason that restricts the development of image description. Compared with the data in the field of natural language processing, which ranges from millions to hundreds of millions, the data sets used in image description research are generally around 100,000. The data scale is too small, resulting in the model usually outputting incoherent or monotonous sentences. For example, the model outputs the incoherent sentence "a bottle of beer sitting on the table".

[0051] The solution of this application is designed and optimized mainly from the following aspects: instead of using the image features output by the detection model, the results of detecting the object name and the detection frame are directly used, and the detected objects are sorted according to prior knowledge, and then intermediate text is generated to directly tell the model where the "key points" of the image are. Integrating a pre-trained language model at the decoding end improves the model's decoding ability.

[0052] Please refer to Figure 3 , which shows a method example diagram of a specific example of a method for improving the image description effect provided by an embodiment of the present invention. This example diagram is only for the convenience of those skilled in the art to better understand the present application and is for reference only. It does not completely correspond to the data of the embodiment of the present application.

[0053] like Figure 3 As shown in the figure, the importance of the detected objects is first evaluated, and the object names are converted into intermediate text according to the importance ranking. The importance is evaluated from three aspects: size, position, and attention probability. In addition, the pre-trained language model is integrated at the transformer decoding end.

[0054] Pay attention to the probability

[0055] The image description training set consists of N images And the corresponding text description The image detection model supports detecting M types of objects. The object list is If an object appears in the image and is also mentioned in the text description, then the object is considered to be attended to. The attention probability is calculated by the following formula:

[0056]

[0057] According to the above formula, the attention probability of all objects supported by image detection is calculated. Assume that the attention probabilities of table, vase and flower are: A 桌子 =0.001, A 瓶子 =0.1, A 鲜花 =0.5.

[0058] 1. Image detection

[0059] Input a picture with a width of W and a height of H, and use the trained image detection model to detect it, and get the name of the object in the picture and the corresponding detection frame [x, y, w, h], where x, y are the coordinates of the center point of the detection frame, w is the width of the detection frame, and h is the height of the detection frame. The image detection model is not described here. The current mainstream image detection models include FastRcnn, Resnet, YOLO, etc. Suppose a picture of size 500*500 is input. After the image detection model, a list of objects in the picture is obtained: [table, bottle, flowers], and the corresponding object detection frame:

[0060] [[300,200,400,300],[280,200,100,200],[200,200,300,200]]

[0061] 2. Order of importance

[0062] 1) Object area score

[0063] The larger the object, the higher the score. The object area score is:

[0064] Area = object area / image area;

[0065] According to the example in 2, the area scores of the table, bottle, and flower are: Area 桌子 =0.48, Area 瓶子 =0.12, Area 鲜花 =0.24.

[0066] 2) Object position score

[0067] The closer the object is to the center of the image, the higher the score. The position score is:

[0068]

[0069] According to the assumption in 2, the position scores of the table, bottle, and flower are: L 桌子 =0.92, L 瓶子 =0.95, L 鲜花 =0.92.

[0070] 3) Total score of object importance

[0071] The object importance score is a fusion of the area score, size score, and attention probability:

[0072] S=αArea+βL+γA

[0073] Among them, α, β, and γ are weights. Let α = β = γ = 1. According to the above example, the importance scores of the table, bottle, and flower are: S 桌子=1.401, S 瓶子 =1.17,S 鲜花 = 1.66. According to the importance, the three objects are ranked as follows: flowers, table, bottle.

[0074] 3. Intermediate text

[0075] Different from the traditional method of generating image descriptions based on image features, this patent expresses the image detection results in text form, reducing the difficulty of training the generation model. The intermediate text can use rules to connect the detected objects through language, such as XX is surrounded by XX. Through the conversion of the intermediate text, the training set of the image description is switched from the original "image->text" to the "text->text" format. According to the importance sorting in 3, the intermediate text examples are listed in a simple way: flowers, tables, bottles. In the training set, the corresponding real image description is: There is a bouquet of flowers on the table.

[0076] 4. Decoding end integrates pre-trained language model

[0077] In order to enhance the network's ability to decode text, a pre-trained language model is integrated at the decoding end. There are two aspects involved here, one is the fusion method, and the other is the selection of the pre-trained language model. There are many pre-trained language models currently, such as BERT, XLNet, etc. There are also many fusion methods in the industry, such as DeepFusion, ColdFusion, etc. This patent uses Cold Fusion as an example of a fusion strategy. Assume that the output state of the decoding end is S, and assume that the output sequence feature of the pre-trained language model is L = [l 1 , l 2 , ..., l M ],in is the language model feature of the character at time t, and d is the output dimension of the language model. The Cold Fusion fusion process is expressed by formula (1-3):

[0078] h L =ReLU(W h L+b h ) (1)

[0079] g L =Sigmoid(W g [S;h L ]+b g ) (2)

[0080]

[0081] in, is the affine transformation parameter, ReLU(·) and Sigmoid(·) are activation functions. Finally, outputE Replace S as the decoding end output state.

[0082] 5. Model Training

[0083] Perform model training based on the model and training data constructed above.

[0084] Please refer to Figure 4 , which shows a block diagram of a device for improving picture description effect provided by an embodiment of the present invention.

[0085] like Figure 4 As shown, the device 400 for improving the picture description effect includes an acquisition program module 410 , a sorting program module 420 and a generation program module 430 .

[0086] Among them, the acquisition program module 410 is configured to obtain the names of all objects in the picture to be described and the detection frames of all the objects; the sorting program module 420 is configured to determine the importance ranking of all the objects at least based on the detection frames of all the objects; the generation program module 430 is configured to generate intermediate text based on the importance ranking and the names of all the objects.

[0087] It should be understood that Figure 4 Modules and references documented in Figure 1 and Figure 2 Therefore, the operations and features described above for the method and the corresponding technical effects are also applicable to Figure 4 The modules in it will not be described in detail here.

[0088] It is worth noting that the modules in the embodiments of the present disclosure are not used to limit the solutions of the present disclosure. For example, the judgment module can be described as a module that judges whether the interactive state is a playback scene when the device is in an interactive state. In addition, the relevant functional modules can also be implemented by a hardware processor, for example, the judgment module can also be implemented by a processor, which will not be repeated here.

[0089] In some other embodiments, the present invention also provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the method for improving the picture description effect in any of the above method embodiments;

[0090] As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows:

[0091] Get the names of all objects in the image to be described and the detection frames of all the objects;

[0092] Determining an importance ranking of all the objects based at least on the detection frames of all the objects;

[0093] An intermediate text is generated based on the importance ranking and the names of all the objects.

[0094] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the device for improving the picture description effect, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device for improving the picture description effect via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any of the above-mentioned methods for improving the picture description effect.

[0096] Figure 5 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the device includes: one or more processors 510 and a memory 520, Figure 5 A processor 510 is taken as an example. The device for improving the image description effect may also include: an input device 530 and an output device 540. The processor 510, the memory 520, the input device 530 and the output device 540 may be connected via a bus or other means. Figure 5 The example of connecting via a bus is taken. The memory 520 is the above-mentioned non-volatile computer-readable storage medium. The processor 510 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 520, that is, the method for improving the picture description effect of the above-mentioned method embodiment is realized. The input device 530 can receive input digital or character information, and generate key signal input related to the user setting and function control of the communication compensation device. The output device 540 may include a display device such as a display screen.

[0097] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0098] As an implementation mode, the electronic device is applied to a device for improving picture description effect, and is used for a client, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can:

[0099] Get the names of all objects in the image to be described and the detection frames of all the objects;

[0100] Determining an importance ranking of all the objects based at least on the detection frames of all the objects;

[0101] An intermediate text is generated based on the importance ranking and the names of all the objects.

[0102] The electronic device of the embodiment of the present application exists in various forms, including but not limited to:

[0103] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0104] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0105] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0106] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0107] (5) Other electronic devices with data interaction functions.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for improving image description effect, comprising: Get the names of all objects in the image to be described and the detection frames of all the objects; Determining an importance ranking of all the objects based at least on the detection frames of all the objects; generating an intermediate text based on the importance ranking and the names of all the objects; The determining the importance ranking of all objects at least based on the detection frames of all objects comprises: Determine an area score of each object based on the size of the detection frame of each object among all the objects occupied by the picture to be described; Determine a position score of each object based on a distance between a detection frame of each object and a center of the picture to be described; Obtaining the attention probability of each object in a preset picture description training set; The importance of all the objects is ranked based on the area score of each object, the position score of each object and the attention probability of each object.

2. The method according to claim 1, wherein: The obtaining of the attention probability of each object in the preset picture description training set comprises: The preset picture description training set includes multiple pictures and a text description corresponding to each picture. If an object appears in a picture in the preset picture description training set, and the object is mentioned in a text description corresponding to a picture in the preset picture description training set, it is determined that the object is being paid attention to, and the attention probability of the object is calculated based on the number of times the object is paid attention to and the number of times the object appears in the preset picture description training set.

3. The method according to claim 1, wherein: After generating the intermediate text based on the importance ranking and the names of all the objects, the method further includes: The intermediate text is input into the picture description model to obtain output picture description text.

4. The method according to claim 3, wherein: The image description model is a transformer model, and the decoding end of the transformer model is fused with a pre-trained language model through a preset fusion method.

5. The method according to claim 4, wherein: The pre-trained language model is BERT or XLNet, and the preset fusion method includes DeepFusion or ColdFusion.

6. The method according to any one of claims 1 to 5, wherein: The intermediate text includes linking all the objects together through relational language according to the importance ranking using preset rules, and the relationship includes a positional relationship.

7. A device for improving picture description effect, comprising: An acquisition program module is configured to acquire the names of all objects in the picture to be described and the detection frames of all the objects; A ranking program module, configured to determine the importance ranking of all the objects based at least on the detection frames of all the objects; A generating program module configured to generate an intermediate text based on the importance ranking and the names of all the objects; Among them, the sorting program module is further configured to: determine the area score of each object based on the size of the detection frame of each object among all the objects in the picture to be described; determine the position score of each object based on the distance between the detection frame of each object and the center of the picture to be described; obtain the attention probability of each object in a preset picture description training set; and sort the importance of all the objects based on the area score of each object, the position score of each object and the attention probability of each object.

8. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 6.

9. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.