Image description generation method and electronic device

Abstract

The present application discloses an image description generation method and electronic device, which relate to the field of image processing technology. After obtaining a first image to be processed, the first image is compared and analyzed with an existing image group to determine the target image group to which the first image belongs; the second image contained in the target image group and the first image are different images collected for the same entity; the initial image description information of the first image and the current image group description information of the target image group are obtained, so as to enhance the initial image description information of the first image based on the current image group description information and obtain the target image description information of the first image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image description generation method and electronic device. Background Art

[0002] A personal knowledge base (PKB) refers to a collection of knowledge collected, organized, managed, and applied by an individual. It usually stores images related to knowledge to help better understand and remember knowledge, so as to effectively store, retrieve, and utilize knowledge resources.

[0003] However, in practical applications, the features contained in the image itself are limited, and may even be the same or similar to the features of similar entity images, which may affect the efficiency and reliability of subsequent access and processing of the image, resulting in failure to meet downstream task requirements. Summary of the invention

[0004] In view of the above problems, this application provides the following solutions:

[0005] The first aspect of the present application provides a method for generating an image description, the method comprising:

[0006] Acquire a first image to be processed;

[0007] Comparing and analyzing the first image with an existing image group, determining a target image group to which the first image belongs; a second image included in the target image group and the first image are different images collected for the same entity;

[0008] Acquire initial image description information of the first image and current image group description information of the target image group;

[0009] The initial image description information is enhanced according to the current image group description information to obtain target image description information of the first image.

[0010] In a possible implementation, the enhancing the initial image description information according to the current image group description information to obtain the target image description information of the first image includes:

[0011] According to the current image group description information, selecting first reference knowledge related to the current image group description information from a knowledge base;

[0012] Obtain corresponding first task prompt information by using the current image group description information, the first reference knowledge, the first image, the initial image description information and the first task prompt word;

[0013] Inputting the first task prompt information into a first model to obtain target image description information of the first image fed back by the first model;

[0014] The first task prompt word is used to guide the first model to compare and analyze the first image and the initial image description information based on the current image group description information and the first reference knowledge to generate the target image description information.

[0015] In a possible implementation, the enhancing the initial image description information according to the current image group description information to obtain the target image description information of the first image includes:

[0016] Obtain corresponding second task prompt information using the target image group, the current image group description information, the first image, the initial image description information, and the second task prompt word;

[0017] Inputting the second task prompt information into the first model to obtain new image description information of the first image fed back by the first model and new image group description information of the target image group;

[0018] According to the new image group description information, selecting second reference knowledge related to the new image group description information from a knowledge base;

[0019] Obtain corresponding third task prompt information using the second reference knowledge, the first image, the new image description information, and the third task prompt word;

[0020] Inputting the third task prompt information into the first model to obtain target image description information of the first image fed back by the first model;

[0021] The second task prompt is used to guide the first model to compare and analyze the first image and the initial image description information according to the current image group description information, and after generating the new image description information, to compare and analyze the target image group and the current image group description information according to the new image description information to generate the new image group description information;

[0022] The third task prompt word is used to guide the first model to compare and analyze the first image and the new image description information based on the second reference knowledge to generate the target image description information.

[0023] In a possible implementation, the enhancing the initial image description information according to the current image group description information to obtain the target image description information of the first image includes:

[0024] Obtain corresponding fourth task prompt information using the current image group description information, the first image, the initial image description information, and a fourth task prompt word;

[0025] Inputting the fourth task prompt information into the second model to obtain target image description information of the first image fed back by the second model;

[0026] The fourth task prompt word is used to guide the second model to compare and analyze the first image and the initial image description information according to the current image group description information to generate the target image description information.

[0027] In a possible implementation, comparing and analyzing the first image with an existing image group to determine the target image group to which the first image belongs includes:

[0028] Performing target entity feature recognition on the first image to obtain a first recognition result;

[0029] If the first recognition result indicates that the first image contains a target entity feature corresponding to any entity, determining the acquisition time corresponding to each of the first image and the second image in the existing image group;

[0030] Determine, according to the acquisition time and the first recognition result, a reference second image in a reference image group of the first image from second images included in an existing image group;

[0031] Comparing the first image with the reference second image for similarity to obtain a second recognition result of the first image; the second recognition result indicates whether the first image and the corresponding reference second image are different images collected for the same entity;

[0032] A target image group to which the first image belongs is determined according to the second recognition result.

[0033] In a possible implementation, the comparing and analyzing the first image with an existing image group to determine the target image group to which the first image belongs further includes:

[0034] If the first recognition result indicates that the first image does not contain a target entity feature corresponding to any entity, cache the first image;

[0035] When determining the image group to which the acquired third image belongs, acquiring similarities between the cached first image and fourth image and the third image respectively;

[0036] determining that the similarity reaches a similarity threshold, and determining the image group to which the third image belongs as the target image group of the first image;

[0037] The third image is an image obtained for the first time after the first image is obtained, and contains target entity features corresponding to any entity;

[0038] The fourth image is at least one image acquired earlier than the first image.

[0039] In a possible implementation, the comparing and analyzing the first image with an existing image group to determine the target image group to which the first image belongs further includes:

[0040] If the first recognition result indicates that the first image does not contain a target entity feature corresponding to any entity, obtaining two reference image groups that are adjacent to the acquisition time of the first image;

[0041] From each of the reference image groups, at least two second images that are adjacent to the first image in acquisition time and contain target entity features of any entity, and at least one second image that does not contain the target entity features of any entity are selected as reference second images;

[0042] performing a similarity comparison between the first image and each of the reference second images in the same reference image group to obtain a corresponding similarity comparison result;

[0043] The target image group to which the first image belongs is determined according to the similarity comparison result.

[0044] In a possible implementation, determining the target image group to which the first image belongs according to the second recognition result includes any one of the following:

[0045] If the second recognition result indicates that the first image and any of the reference second images are different images collected for the same entity, determining the reference image group where the reference second image belongs as the target image group of the first image;

[0046] If the second recognition result indicates that the first image and each of the reference second images are not different images collected for the same entity, the first images constitute a new image group; the new image group is determined as the target image group to which the first image belongs;

[0047] Wherein, when the target image group only includes the first image, the first image is analyzed to obtain the target image description information of the first image.

[0048] In a possible implementation, the acquiring initial image description information of the first image and current image group description information of the target image group includes:

[0049] Using the first image and the fifth task prompt word, obtaining corresponding fifth task prompt information;

[0050] Inputting the fifth task prompt information into the third model to obtain initial image description information of the first image fed back by the third model;

[0051] Using the target image group, image description information of each image included in the target image group and the sixth task prompt word, corresponding sixth task prompt information is obtained, the sixth task prompt information is input into the third model, and current image description information of the target image group fed back by the third model is obtained; or the stored current image group description information of the target image group is read;

[0052] The fifth task prompt word is used to guide the third model to analyze the first image and generate the initial image description information;

[0053] The sixth task prompt word is used to guide the third model to analyze each image included in the target image group and the image description information of the image, and generate current image description information of the target image group;

[0054] The read current image group description information is generated before acquiring the first image based on the image description information of the second image or third reference knowledge related to the target image group.

[0055] A second aspect of the present application provides an electronic device, the electronic device comprising: at least one communication element, at least one memory and at least one processor, wherein:

[0056] The communication element is used to receive a first image to be processed;

[0057] The memory is used to store a plurality of computer instructions;

[0058] The processor is used to load and execute the multiple computer instructions to implement the following steps:

[0059] acquiring the first image;

[0060] Comparing and analyzing the first image with an existing image group, determining a target image group to which the first image belongs; a second image included in the target image group and the first image are different images collected for the same entity;

[0061] Acquire initial image description information of the first image and current image group description information of the target image group;

[0062] The initial image description information is enhanced according to the current image group description information to obtain target image description information of the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0064] Figure 1 A schematic diagram of a flow chart of an image description generation method proposed in Embodiment 1 of the present application;

[0065] Figure 2 A schematic diagram of an initial image description information enhancement effect in an image description generation method proposed in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of a flow chart of an image description generation method proposed in Embodiment 2 of the present application;

[0067] Figure 4 A schematic diagram of another initial image description information enhancement effect in an image description generation method proposed in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of a flow chart of an image description generation method proposed in Embodiment 3 of the present application;

[0069] Figure 6 A schematic diagram of a flow chart of an image description generation method proposed in Embodiment 4 of the present application;

[0070] Figure 7 A schematic diagram of image grouping for an image description generation method provided in an embodiment of the present application;

[0071] Figure 8 A schematic diagram of a scene of an image description generation method provided in an embodiment of the present application;

[0072] Fig. 9 A schematic diagram of the structure of an image description generating device provided in an embodiment of the present application;

[0073] Fig.10 The figure is a schematic diagram of the hardware structure of an electronic device suitable for the image description generation method proposed in the present application. DETAILED DESCRIPTION

[0074] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0075] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0076] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0077] Regarding the content described in the background technology section, with the rapid development and application of artificial intelligence technology, it is proposed to input each image into a generative model based on artificial intelligence to generate a natural language description of the objects and actions in the image, namely, image description information. However, for different images of similar entities, it is easy for the generated image description information to be the same because the image features they contain are the same or similar, which will still affect the efficiency and reliability of image access processing, thereby affecting the normal and reliable execution of downstream tasks.

[0078] In order to solve the above problems, the embodiment of the present application provides an image description generation method. For the acquired image to be processed (for the convenience of description, it is recorded as the first image), the target image group to which it belongs is determined, and the image group description information of the target image group is obtained. After the initial image description information of the first image is obtained, the present application proposes to enhance the initial image description information of the first image based on the image group description information. At this time, since the same image group contains different images collected for the same entity, the image group description information of the image groups of different entities is different, that is, the image group description information used for the initial image description information enhancement of different images of similar entities is different, which ensures that the enhanced target image description information is different, and more accurately and comprehensively describes the visual content, spatial relationship and semantic information in the first image, and can also provide contextual background information to improve the image enhancement effect, thereby improving the accessibility of the first image, and can better meet the needs of downstream tasks (such as cross-modal understanding / search, visual question answering or image-text matching, etc.), and improve task performance. The image description generation method of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0079] Reference Figure 1 , is a flow chart of an image description generation method proposed in the first embodiment of the present application, which method can be applied to an electronic device, which can be a terminal device or a server, such as an application with data processing capability running on the terminal device or a cloud service provided by a cloud server. Figure 1 As shown, the image description generation method provided in this embodiment may include but is not limited to the following steps:

[0080] Step S11, obtaining a first image to be processed;

[0081] In the embodiment of the present application, the first image can be any image directly captured on site by an image acquisition device (such as a mobile phone, a camera, a video recorder, or other terminal devices with a camera, etc.). Afterwards, if the image acquisition device itself is a terminal device with data processing capabilities, the subsequent steps can be directly executed, or sent to other local terminal devices with data processing capabilities (such as a laptop or desktop computer, etc.), or uploaded to a cloud server through the network to execute subsequent steps. It can be seen that in this scenario, the first image acquired in step S11 can be a directly captured image or a captured image sent by the image acquisition device.

[0082] Of course, in actual applications, the first image may also be obtained by the electronic device from a web page, storage device, database, or other third-party platform. This application does not limit the method of obtaining the first image, which can be determined based on actual conditions.

[0083] Step S12, performing comparative analysis on the first image and the existing image group to determine the target image group to which the first image belongs; the second image and the first image included in the target image group are different images collected for the same entity;

[0084] In this application, different images of the same entity are assigned to an image group. The entity can be a thing or concept such as a car brand, a conference / speech topic, or a scenic spot name. It can be determined based on the actual scene in which the image is acquired. This application does not limit the content of each entity. Therefore, different images of the same entity can be images collected for the same entity in the same scene under one or more combinations of acquisition attributes such as different viewing angles / scene positions / entity features / times / camera configuration parameters, or images collected for the same entity in different scenes (or under one or more combinations of these acquisition attributes).

[0085] According to the above analysis, in the embodiment of the present application, different entities corresponding to different image groups can be entities of the same type with different manufacturers or different brands. For example, for an entity of the type of car, different images of cars of the same brand can be assigned to one image group; for an entity of the type of meeting (or speech), different images of the same meeting (or speech) theme / name can be assigned to one image group; for an entity of the type of scenery, different images of the same scenic spot name can be assigned to one image group, etc. The present application does not limit the method for obtaining each image group by grouping images. According to actual needs, image grouping can also be combined with one or more themes such as scenery, people, events, emotions or uses, etc. The implementation process is not described in detail in the present application.

[0086] Based on this, when at least one image group has been stored in the database of the electronic device or its communication, the currently acquired first image can be compared and analyzed with the existing image groups. If the entity of the second image already in a certain image group is the same as the entity in the first image, the image group is determined as the target image group to which the first image belongs. The present application does not limit the implementation method for determining the target image group. The first image and the existing image groups can be compared and analyzed in combination with the grouping features corresponding to the image grouping method described above but not limited to the above. The implementation process is not described in detail in this application.

[0087] Step S13, obtaining initial image description information of the first image and current image group description information of the target image group;

[0088] In the actual application of the present application, in a possible implementation, the initial image description information of the first image is a natural language description converted from the image content of the first image itself, and an image description generation technology combining the field knowledge of computer vision and natural language processing can be used to create text that can describe the scene, object and action in the first image. For example, the first image is input into any target model such as a generative model, a multimodal large language model (MLLM), a visual large language model VLLM (VsionLLM), a visual multimodal large model or a GPT (Generative Pre-trained Transformer) model (such as GPT-3 or GPT-4) or other general large models for image analysis to generate an accurate and vivid text description based on the image content as the initial image description information of the first image. Similarly, the current image group description information of the target image group can also be generated in this way, such as inputting each image contained in the target image group into any target model as above to generate the corresponding current image group description information.

[0089] In another possible implementation, step S13 can also obtain the initial image description information of the first image in response to the user's description editing operation on the first image. Similarly, after the target image group is determined for the first time, that is, the image group in which the second image is located is determined for the first time, the user can view the second images contained in the image group, as well as attribute information such as the name of the image group, and obtain the current image description information of the target image group in response to the user's description editing operation on the target image group. The present application does not describe the image description editing implementation process in detail.

[0090] Of course, the present application may also adopt other image description generation methods to obtain the initial image description information of the first image, and the implementation process is not described in detail one by one in the present application. For the current image group description information of the target image group, when the last of the second images included in the current target image group is assigned to the target image group, the image group description information of the target image group may be generated according to the method described above, but not limited to it, or the image description information of the target image group before the assignment is updated (which may be described in the corresponding part of the context embodiment) as the current image group description information of the target image group, and at this time, step S13 may directly read the configured image group description information of the target image group as the current image group description information.

[0091] Step S14: enhancing the initial image description information of the first image according to the current image group description information to obtain the target image description information of the first image.

[0092] Following the above analysis of the technical solution of the present application, and the generation process of the initial image description information of the first image described, only the characteristic content of the first image itself is considered in the generation process of the initial image description information, and the information content matches the image data of the first image itself one by one. In this regard, the present application proposes to use the image group description information of the target image group to which the first image belongs to enhance the initial image description information of the first image, such as enhancing the information in the initial image description information that is beneficial to image recognition and suppressing the information that is not beneficial to image recognition, so that the enhanced target image description information can capture the visual content, spatial relationship, semantic information and contextual background of the image more accurately and finely, and improve the access processing accuracy of the first image, so as to use the target image description information to better meet the downstream task requirements.

[0093] For example, the image description generation method proposed in this application is executed on an image captured in a car show scene as an example. Figure 2 As shown in FIG. (a), the initial image description information of the first image is obtained according to the method described above but not limited thereto. After determining the target image group to which the first image belongs, such as an image group consisting of the second images related to the cars of Company A, the initial image description information of the first image is enhanced according to the description information of the current image group, and the following can be obtained: Figure 2 The target image description information of the first image shown in Figure (b) is shown in the figure. Obviously, the target image description information adds information relative to the initial image description information, which more comprehensively and finely describes the impact / significance of the first image content in the automotive industry and Company A, and at the same time promotes Company A, which can better meet the needs of downstream tasks. It should be noted that the target image description information content and its output method include but are not limited to Figure 2 The method shown in Figure (b).

[0094] For the enhancement process of step S14, in a possible implementation, the initial image description information of the first image can be obtained in combination with the current image group description information as the background knowledge for the target model to generate the image description, so as to generate the supplementary description information for the initial image description of the first image, and fuse the supplementary description information with the initial image description information (such as splicing) to obtain the target image description information. Optionally, the target model can also understand / analyze the image content of the first image more comprehensively and accurately based on the current image group description information and the initial image description information, and regenerate the image description information of the first image as the target image description information. Or the target model reflects on the initial image description information of the first image based on the current image group description information, checks whether it accurately reflects the image content, whether there are omissions or errors, and then adjusts the initial image description information based on the inspection results to generate higher quality target image description information for the first image.

[0095] In a possible implementation, the present application may also fine-tune the target model based on the current image group description information, so that the obtained model can understand and generate descriptions consistent with the current image group description information, improve the accuracy and relevance of the image description, and then re-input the first image into the fine-tuned target model, or input the first image and its initial image description information into the fine-tuned target model, reflect and integrate the initial image description information based on the image content of the first image, and use the integrated initial image description information as the target image description information of the first image, etc. The present application does not limit how to implement step S14 based on the target model. Of course, step S14 can also be implemented using other types of image description generation technology or information enhancement technology.

[0096] In some embodiments, the manual editing method described above may be used to implement step S14. In this case, the current image group description information and the initial image description information of the first image may be output, and in response to the enhancement operation on the initial image description information, the target image description information of the first image is obtained. The present application does not limit the implementation method of step S14.

[0097] In summary, in the embodiment of the present application, a first image to be processed is obtained, and after determining the target image group to which the first image belongs through comparative analysis with an existing image group, in order to improve the accessibility of the first image, the initial image description information of the first image is obtained to express the content of the first image. Furthermore, the present application will also enhance the initial image description information based on the current image group description information of the target image group, so that the content of the obtained target image description information is richer and more comprehensive, reflecting the entity-related content captured by the first image, such as Figure 2 In the example, the car-related content of Company A supplements or replaces the image data of the first image itself. In particular, for multiple images containing different entities of the same type, such as images of cars from different companies, since these multiple images belong to different target image groups, even if their initial image description information is similar, after the enhanced processing of the target image group description information, the respective target image description information is guaranteed to be different, so that the target image description information of a single image can be used to better meet the requirements of downstream tasks such as cross-modal understanding / search, visual question answering, or image-text matching, thereby improving the efficiency, accuracy, and reliability of image access processing.

[0098] Reference Figure 3 , is a flow chart of an image description generation method proposed in the second embodiment of the present application. This embodiment can describe an optional enhancement processing method for the initial image description information of the first image. For other steps of the image description generation method, reference can be made to the description of the corresponding part of the above embodiment, and this embodiment will not be repeated. Figure 3As shown, the optional implementation process of step S14 in the above embodiment may include:

[0099] Step S31, based on the current image group description information of the target image group, first reference knowledge related to the current image group description information is selected from the knowledge base;

[0100] In order to obtain a deeper understanding of the first image, a richer contextual background and abstract concepts, the present application proposes to use the relevant information of the first image and / or its initial image description information to enhance the initial image description information of the first image. To this end, a retriever of the RAG (Retrieval-Augmented Generation) technical architecture can be used to retrieve reference knowledge related to the first image and / or its initial image description information from the knowledge base, such as documents contained in the knowledge base whose similarity with the first image and / or its initial image description information is greater than a similarity threshold (such as 80% or 90%, etc., and this application does not limit its value), so as to implement the initial image description information of the first image in an enhanced manner according to the context description. It should be noted that the retrieval of reference knowledge is not limited to the implementation of the RGA technology, but can also be implemented directly based on the image-text matching algorithm. The implementation process is not described in detail in this application.

[0101] Preferably, since the first image contains limited content and may not contain the significant features of the corresponding entity, the reference knowledge retrieved related to the first image is not accurate enough and cannot achieve the purpose of enhancing the initial image description information. In this regard, the present application proposes to screen various reference knowledge related to the current image group description information based on the current image group description information of the target image group to which the first image belongs, and record them as first reference knowledge, so as to enrich the first reference knowledge and ensure that it is information related to the entity and scene described in the first image, so as to lay a reliable foundation for the subsequent enhancement of the initial image description information of the first image.

[0102] Among them, the screening process of the first reference knowledge can be implemented according to the RGA technology, or according to various image-text matching algorithms, and the present application does not limit the screening method of step S31. It should be noted that the screened knowledge base can include one or more of various open source knowledge bases (including databases and libraries disclosed on the network platform, etc.), corporate knowledge bases, and personal knowledge bases, and the present application does not limit the type of knowledge base and its source.

[0103] Step S32, obtaining corresponding first task prompt information by using the current image group description information, the first reference knowledge, the first image, the initial image description information of the first image, and the first task prompt word;

[0104] Step S33, inputting the first task prompt information into the first model to obtain the target image description information of the first image fed back by the first model; wherein the first task prompt word is used to guide the first model to compare and analyze the first image and the initial image description information based on the current image group description information and the first reference knowledge to generate the target image description information.

[0105] Following the above analysis, the present application can enhance the initial image description information based on the current image group description information and its related first reference knowledge to obtain the target image description information of the first image. Figure 2 The image description enhancement of the first image captured in the car show scene shown in FIG. Figure 4 The first image and its initial image description information shown in FIG. 1 (a) can be obtained by using but not limited to the enhancement process described in the corresponding part of the above embodiment 1. Figure 4 The target image description information of the first image shown in (b) is relative to Figure 2 As shown in Figure (b), the enhancement result (target image description information) of the initial image description information using the current image group description information, the first reference knowledge related to the current image group description information retrieved in this embodiment usually includes content that is not included in the image data of each second image in the target image group itself, enriching the contextual background and abstract concept of the first image, so that it can achieve better performance in downstream tasks.

[0106] For the enhancement of the initial image description information, preferably, the present application can use any of the target models listed above as the first model, such as the visual large language model VLLM, or the generation model under the RAG architecture, to achieve the enhancement processing of the initial image description information of the first image. In this implementation process, in one possible implementation, a corresponding prompt word (Prompt, denoted as the first task prompt word) can be pre-configured for the image description enhancement task to guide the first model on how to perform the image description enhancement task. This embodiment can be to guide the first model to compare and analyze the first image and the initial image description information based on the current image group description information and the first reference knowledge to generate the target image description information of the first image.

[0107] Therefore, the first task prompt word of the present application may include instructions for the processing logic of the first model to generate target image description information of the first image, and may also indicate the task requirement information of the image description enhancement task in which the first model enhances the initial image description information of the first image based on the current image group description information and its related first reference knowledge, so as to help the first model more accurately capture the intent of the task, so as to generate an output that is more in line with expectations, namely, the target image description information, etc. The present application does not impose any restrictions on the content of the first task prompt word.

[0108] In practical applications, the first task prompt word can be a prompt word template, such as an image description enhancement template, which can be adjusted and customized according to actual needs to adapt to different scenarios and purposes, and improve the efficiency and effect of interaction with the first model. The present application queries the configured image description enhancement template (i.e., the first task prompt word), configures the corresponding field content of the image description enhancement template based on the acquired current image group description information and its related first reference knowledge, as well as the first image and its initial image description information, and records the configured image description enhancement template as the first task prompt information and inputs it into the first model. The first model can parse the first task prompt information, identify the contents of each field contained therein, and then compare and analyze the first image and the initial image description information according to the processing logic / processing requirements represented by the first task prompt word based on the current image group description information and the first reference knowledge, and generate the target image description information, i.e., enhance the initial image description information of the first image. The enhancement implementation process can refer to the description of the corresponding part of the above embodiment, and this embodiment will not be described in detail here.

[0109] It should be understood that in the process of enhancing the initial image description information of the first image through the first model, in order to improve the enhancement effect, the first model can use the first reference knowledge and the current image group information as background knowledge to reflect on the first image and / or the initial image description information, and obtain the target image description information after multiple optimization and integration of the initial image description information. In each reflection and integration process, the corresponding integrated image description information can be output, and the user can verify whether it is appropriate. If it is not appropriate, the corresponding adjustment instructions (which can be used as new task prompts) are continuously input to the first model to guide the first model to continue to reflect and integrate the first image and / or the image description information, and so on, after multiple interactions, until the target image description information of the first image is obtained, that is, the image description information that meets the actual needs.

[0110] It can be seen that in the embodiment of the present application, after obtaining the initial image description information of the first image and the current image group description information of the target image group to which the first image belongs, the first reference knowledge related to the current image group description information will be further retrieved to enhance the initial image description information, improve the image description enhancement efficiency, and obtain high-quality target image description information. In addition, during the enhancement process, the present application directly utilizes the powerful ability of the first model in image description enhancement. It only needs to input the various input information required to enhance the initial image description information into the first model, and the first model automatically completes the enhancement processing of the initial image description information, and efficiently and reliably generates more detailed and accurate target image description information of the first image, so as to use the target image description information to better meet the needs of various downstream tasks, so that the downstream tasks are no longer limited to the first image, and the performance of downstream tasks is improved.

[0111] Reference Figure 5 , is a flow chart of an image description generation method proposed in Example 3 of the present application. This embodiment can describe another optional enhancement processing method for the initial image description information of the first image. Compared with the image description enhancement processing method described in Example 2, this embodiment can first use the current image group description information of the target image group to which the first image belongs to enhance the initial image description information of the first image, and after obtaining the new image description information of the first image, use the new image description information to update the current image group description information to obtain the new image group description information of the target image group (which can be used as the current image group description information of the target image group when the initial description information of the next first image is enhanced), and then retrieve reference knowledge related to the new image group description information (recorded as the second reference knowledge), enhance the above-mentioned new image description information, and obtain the target image description information of the first image.

[0112] It can be seen that the second reference knowledge retrieved by the new image group description information in this embodiment is usually more accurate than the first reference knowledge retrieved in the second embodiment, and the enhancement effect of the image description information of the first image is better, and higher quality target image description information can be obtained. Based on this, when the model is used to implement the image description information enhancement processing, the other steps of the image description generation method proposed in this application described in the first embodiment above are combined. Figure 5 As shown, the optional implementation process of step S14 in the above embodiment may include:

[0113] Step S51, using the target image group to which the first image belongs and its current image group description information, the first image and its initial image description information, and the second task prompt word, to obtain corresponding second task prompt information;

[0114] Step S52, inputting the second task prompt information into the first model, obtaining new image description information of the first image fed back by the first model, and new image group description information of the target image group;

[0115] In the implementation of the present application, the second task prompt word can be a text used to guide the first model to compare and analyze the first image and the initial image description information based on the current image group description information, generate new image description information of the first image, and then compare and analyze the target image group and the current image group description information based on the new image description information to generate new image group description information of the target image group. The configuration method and expression form of the second task prompt word can be the same as the first task prompt word, such as a prompt word template, so as to write the target image group to which the first image belongs and its current image group description information, the first image and its initial image description information into the corresponding fields of the prompt word template to obtain the second task prompt information.

[0116] Optionally, the second task prompt word may also be a plurality of instructions or task processing logic information that guide the first model to perform the above-mentioned task, so that after receiving the second task prompt word, the first model can enhance the initial image description information and the image group description information according to the guidance of the second task prompt word. Regarding the generation process of the new image description information of the first image, the implementation method of using the current image group description information to enhance the initial image description information and obtain the target image description information can refer to the one described in Example 1. This process can be integrated through at least one reflection to generate more detailed image description information. This embodiment does not describe the implementation process in detail.

[0117] Regarding the process of generating the new image group description information of the target image group, since the first image is a new image assigned to the target image group, the number of images included in the target image group increases. According to the first generation method of its image group description information, combined with the new image description information of the newly added first image, the second images and the first images included in the current target image group and their corresponding image description information can be reflected and integrated to generate more accurate image group description information, which is recorded as the new image group description information. The reflection and integration process can refer to the description of the corresponding part of the above embodiment 2, and this embodiment is described in detail here.

[0118] In a possible implementation, during the implementation of step S52, the second task prompt word can be split into two sub-task prompt words of corresponding structures, namely, a first sub-task prompt word for guiding the first model to compare and analyze the first image and the initial image description information based on the current image group description information to generate new image description information of the first image, and a second sub-task prompt word for guiding the first model to compare and analyze the target image group and the current image group description information based on the new image description information of the first image to generate new image group description information of the target image group.

[0119] In this case, the current image group description information, the first image and its initial image description information, and the first subtask prompt word can be first input into the first model to obtain new image description information generated by the first model, and then the new image description information, the target image group and its current image group description information, and the second subtask prompt word can be input into the first model to obtain new image group description information generated by the first model. In this implementation process, the corresponding subtask prompt information can also be formed according to the method described above and then input into the first model.

[0120] Step S53, based on the new image group description information, filtering second reference knowledge related to the new image group description information from the knowledge base;

[0121] Regarding the screening process of the second reference knowledge, reference may be made to the screening method of the first reference knowledge in the above second embodiment, which will not be elaborated in this embodiment.

[0122] Step S54, using the second reference knowledge, the first image and its new image description information, and the third task prompt word to obtain corresponding third task prompt information;

[0123] Step S55, input the third task prompt information into the first model to obtain the target image description information of the first image fed back by the first model; the third task prompt word is used to guide the first model to compare and analyze the first image and the new image description information based on the second reference knowledge to generate the target image description information.

[0124] Among them, the configuration method and expression form of the third task prompt word can be the same as the first task prompt word, such as a prompt word template, multiple instructions or task processing logic information that guide the first model to perform the above-mentioned task (such as the task content described in step S55), etc. This application does not elaborate on the third task prompt word and its corresponding third task prompt information acquisition process.

[0125] According to the above analysis, compared with the initial image description information enhancement method described in Example 1, the embodiment of the present application proposes to implement enhancement processing in combination with relevant reference knowledge to improve the enhancement effect, and in the reference knowledge retrieval process, compared with the retrieval of reference knowledge related to the initial image description information of the first image or the image group description information of the target image group described in Example 2, this embodiment proposes to first reflect on and integrate the image description information of the target image group and the first image to improve the accuracy and comprehensiveness of the image / image group description information, and then screen the second reference knowledge related to the new image group description information of the target image group to improve the screening quality, thereby laying a solid foundation for the subsequent effective enhancement of the image description information of the first image.

[0126] In the process of comparing and analyzing other input information according to the input task prompt words by the first model, such as the implementation process of step S55, compared with using the first model's own knowledge base to understand the first image and the target image group, the present application uses the second reference knowledge retrieved from the external knowledge base as background knowledge to reflect on the first image and the target image group, thereby achieving a deeper understanding of the first image and the target image group and improving the accuracy and comprehensiveness of the target image description information generated therefrom. Of course, the image description information of the first image finally generated also enhances the understanding of the second reference knowledge to improve the performance of downstream tasks performed using the second reference knowledge.

[0127] In combination with the above analysis, in some other embodiments proposed by the present application, the present application may also directly use the image group description information of the target image group to enhance the initial image description information of the first image. In this case, it is not necessary to combine the relevant reference knowledge in the external knowledge base, and a second model may be used for implementation. The second model may not use the RAG architecture, and may be a VLLM model, etc. Based on this, the optional implementation process of the above step S14 may include the following steps:

[0128] The corresponding fourth task prompt information is obtained by using the current image group description information of the target image group to which the first image belongs, the first image, the initial image description information, and the fourth task prompt word, and the fourth task prompt information is input into the second model to obtain the target image description information of the first image fed back by the second model. The fourth task prompt word is used to guide the second model to compare and analyze the first image and the initial image description information according to the current image group description information to generate the target image description information.

[0129] Among them, the configuration method and expression form of the fourth task prompt word can be the same as the first task prompt word, such as the prompt word template, multiple instructions guiding the first model to perform the above task or task processing logic information, etc., which can be the same or similar to the above-mentioned first subtask prompt word. This application does not elaborate on the fourth task prompt word and its corresponding fourth task prompt information acquisition process.

[0130] In summary, the above embodiments describe various image description enhancement methods. According to actual conditions, the image group description information of the target image group to which the first image belongs and / or its related reference knowledge can be used to enhance the initial image description information of the first image, and the obtained target image description information can be used to implement downstream tasks. Compared with directly using the initial image description information to implement downstream tasks, the target image description information not only captures the visual content, spatial relationship and semantic information of the first image, but also provides contextual background and abstract concepts, thereby greatly improving the performance of downstream tasks.

[0131] In conjunction with the above description of using the target model to generate the new / target image description information of the first image and the new image group description information of the target image group, in some embodiments, the implementation process of the above step S13 using a target model (referred to as the third model) may include but is not limited to the following steps:

[0132] Using the first image and the fifth task prompt word, the corresponding fifth task prompt information is obtained, and the fifth task prompt information is input into the third model to obtain the initial image description information of the first image fed back by the third model; the fifth task prompt word is used to guide the third model to analyze the first image and generate the initial image description information.

[0133] It can be seen that the fifth task prompt word can be a prompt word template for generating an image description, which can include how to analyze the content of the first image and use the information obtained from the analysis to generate processing logic or instructions for initial image description information, so as to guide the third model to perform the image description generation task. Of course, the fifth task prompt word can also be an image description generation instruction, which notifies the third model to generate image description information for the input first image. At this time, the third model will analyze the first image through its own image description generation capability and generate initial image description information for the first image.

[0134] In the process of acquiring the current image group description information of the target image group, in a possible implementation, the target image group, the image description information of each image contained in the target image group (which may be the target image description information corresponding to each second image, and the initial image description information of the first image may be added as needed) and the sixth task prompt word may be used to obtain the corresponding sixth task prompt information, and the sixth task prompt information is input into the third model to obtain the current image description information of the target image group fed back by the third model, which may be used as the initial image group description information of the target image group, and may be used to enhance the initial image description information directly according to the method described above, or may be used to enhance the initial image description information after enhancing and generating new image group description information.

[0135] Among them, the sixth task prompt is used to guide the third model to analyze each image contained in the target image group and the image description information of the image, and generate the current image group description information of the target image group. The implementation process can refer to the relevant description content of the first model and the second model in the above embodiment to generate the image group description information, and this application will not be described in detail here. It should be understood that the third model of the present application can be a VLLM model or other trained image description generation model, etc. The present application does not limit the type of the third model. The various models involved in this application can be a general large model or a special large model that is fine-tuned based on task requirements. The embodiments of this application do not limit this.

[0136] In a possible implementation, each second image in the target image group is used as the first image to be processed, and in executing the image description generation method proposed in the present application, the initial / new image group description information of the target image group is generated according to the method described in the above embodiment, which can be stored as the current image group description information of the target image group. In this way, in the process of executing the image description generation method on the new first image acquired this time, after determining the target image group to which the first image belongs, the current image group description information of the stored target image group can be directly read without using the third model to regenerate it. It should be noted that the read current image group description information is generated before acquiring the first image based on the image description information of the second image or the third reference knowledge related to the target image group, and the implementation process can refer to the description of the corresponding part of the above embodiment.

[0137] In the image description generation method described in the above embodiments, the image grouping method may include but is not limited to grouping according to themes such as scenery, people, events, emotions or purposes, time or location and other acquisition attributes, and assigning the acquired images into different image groups. The entities contained in each image in different image groups are different, and the same image group contains different images for the same entity. The target image description information of each image obtained according to the image description generation method proposed in this application and the current image group description information of each image group can be combined to form a personal knowledge base (i.e., a multimodal knowledge base) to help individuals store, retrieve and utilize knowledge resources more effectively, and better meet the downstream task requirements based on the knowledge base. For example, when viewing a speech paper, the relevant images taken by the user at the speech can be automatically retrieved. For example, when selecting a car model, the relevant images of the entity of the car model taken by the user can be automatically retrieved, without the need for manual query by the user, thereby improving the user experience.

[0138] However, grouping images by time or geographic location as described above cannot effectively identify fine-grained topics, such as multiple speeches at the same venue or multiple car models at the same exhibition, which cannot be accurately distinguished by time or geographic location, resulting in inaccurate image grouping. Similarly, when obtaining the target image group to which the first image belongs according to this method, the obtained target image group may be inaccurate, which in turn leads to the inability to enhance the initial image description information of the first image based on its image group description information, and may even reduce the accuracy and reliability of the processed image description information, thereby reducing the performance of downstream tasks performed accordingly.

[0139] Furthermore, grouping images based on image entity features or similarities may confuse photos of cars of different brands with similar physical appearance (especially when the car logo is not captured) or photos of different speeches with similar backgrounds, resulting in inaccurate image grouping. Similarly, the target image group obtained thereby will be incorrect, and the purpose of enhancing the initial image description information of the first image cannot be achieved.

[0140] In order to improve the above problems, the present application proposes a new image grouping method, which adopts an entity fine-grained classification image method, and based on this, realizes a new method of obtaining the target image group to which the first image belongs. The time series can be introduced into the image, and the contextual connection between different images can be established. Through significant feature detection and content similarity analysis, image grouping and determination of the target image group to which the first image belongs are realized, so that an image group can fully present the overall picture of the same entity, such as fully presenting all aspects of the vehicle model, enhancing the semantic understanding of the image and the organization of knowledge, facilitating the association of topics of different modalities, laying the foundation for the subsequent reliable enhancement of the initial image description information of the first image, and improving the image description enhancement effect.

[0141] Based on this, refer to Figure 6 , is a flowchart of an image description generation method proposed in Embodiment 4 of the present application. This embodiment can describe an optional implementation method of how to determine the target image group to which the first image group belongs in the image description generation method proposed in any of the above embodiments. For other steps of the image description generation method, reference can be made to the description of the corresponding parts of the above embodiments, and this embodiment will not be repeated. Based on this, Figure 6 As shown, an optional implementation method of the above step S12 may include but is not limited to the following steps:

[0142] Step S61, performing target entity feature recognition on the first image to be processed to obtain a first recognition result;

[0143] Step S62, determining whether the first image contains a target entity feature corresponding to any entity according to the first recognition result; if yes, proceeding to step S63; if no, executing step S67;

[0144] In the embodiment of the present application, the target entity feature of any entity can be a significant feature / core entity feature of the entity, that is, a feature that can clearly identify the entity, that is, an entity feature that the entity has that can distinguish other entities, and the target entity feature can be used to accurately identify the unique corresponding entity. For example, in the process of identifying entities in an image, based on the entity features extracted from the image, after determining the predicted probability / score (which can be called confidence) that the entity in the image belongs to a known type of entity, the type of entity with the highest confidence is determined as the entity identified from the image. Then, the target entity feature is an entity feature that can obtain a high confidence, which can be called a high-confidence entity feature. If the image contains the target entity feature, it can be quickly and accurately determined that the image contains an entity corresponding to the target entity feature.

[0145] Exemplarily, for entities such as cars, the target entity feature may be an obvious car brand logo or car name in the image; for entities such as speeches, the target entity feature may be the title or name of the speech contained in the image; for landscape entities, the target entity feature may be a specific scenic spot name, etc. In this way, for images containing such target entity features, it is indicated that the image is an image of the corresponding entity, and it can be assigned to the image group for the entity. It should be understood that the acquired images do not necessarily contain target entity features such as car brand logos, speech titles, etc. For images that do not contain such target entity features, entity recognition of such images often results in a relatively low confidence level that the images belong to a certain entity. In order to improve the accuracy of image grouping, further analysis is needed. The implementation process can refer to the description of the corresponding part of the embodiment below.

[0146] Based on the above analysis, in order to improve the efficiency and accuracy of image entity recognition, the present application can, after obtaining an image to be processed (such as the first image mentioned above), pre-determine the target entity features of the entity in the image or its scene, or the target entity features of multiple entities that may be involved, and record the correspondence between different entities and their corresponding target entity features (which can be the high-confidence entity features mentioned above) so as to subsequently realize entity recognition of the image to be processed.

[0147] Among them, in order to clearly determine whether the entity features of the image are target entity features or other entity features, it can be pre-specified that when any pre-configured target entity feature is recognized, a higher prediction probability / score, such as a score greater than 0.9, is output; otherwise, a lower prediction probability / score, such as a score less than 0.5, is output, so as to directly determine whether the input image contains the target entity feature.

[0148] In combination with the above analysis, in a possible implementation, if a fourth model such as a VLLM model is used to implement step S61, the pre-configured sixth task prompt word may be used to guide the fourth model to identify the target entity features in the input image, and output the predicted probability / score corresponding to the identified entity features to determine that the image belongs to the corresponding entity. In this regard, the sixth task prompt word may include the pre-configured target entity features corresponding to different entities, and the minimum / maximum predicted probability / score that needs to be output in different situations of whether the entity features actually identified are the target entity features.

[0149] In this way, the first image and the sixth task prompt word are input into the fourth model, and the first recognition result of the first image obtained can indicate whether the first image contains the target entity feature corresponding to any entity. Among them, the first recognition result may include the predicted probability / score corresponding to the recognized entity feature. If it does not reach the minimum predicted probability / score corresponding to the target entity feature, it means that the first image contains the target entity feature; otherwise, it means that the first image does not contain the target entity feature. Optionally, the first recognition result may also directly mean that the first image contains a high-confidence entity feature (i.e., the target entity feature), or that the first image does not contain a high-confidence entity feature. This application does not limit the content of the first recognition result.

[0150] Combined with the above analysis, illustratively, the sixth task prompt word guides the fourth model to complete the following tasks for the input image:

[0151] Task 1, point out the target entity features of the input image: for example, if the input image shows an image of a car, the target entity features are the brand and model of the car; if the input image is an image of a conference speech, the target entity features are the name of the conference (speech); if the input image is a landscape image, the target entity features are the name of the scenic spot, etc. Here, we can list the target entity features corresponding to various known entity images.

[0152] Task 2: Output the degree to which the recognized entity features belong to the target entity features (i.e., the confidence level of the input image that corresponds to the entity), ranging from 0.0 to 1.0. If the recognized entity features are of a specific brand or a specific speech topic, please output a score greater than 0.9. If the recognized entity features are not specific and the corresponding entity is relatively broad and general, such as "car central control", please output a score lower than 0.5.

[0153] Task 3: Output the first recognition result in the following format:

[0154] Target entity feature: xxxx

[0155] The degree of target entity characteristics: xxxx.

[0156] Based on this, taking the example of Task 2, the first image is an image containing the logo of the automobile brand of Company B, and the first recognition result of the first image output by the fourth model can be:

[0157] Target entity characteristics: B company brand car;

[0158] Degree of target entity features: 0.95.

[0159] If the first image includes an image of a car's central control panel, the first recognition result of the first image output by the fourth model may be:

[0160] Target entity features: the vehicle displayed on the car’s central control screen;

[0161] Degree of target entity feature: 0.5.

[0162] It should be noted that the content of the first recognition result output by the fourth model and its output form include but are not limited to the above example method. According to actual needs, the content of the sixth task prompt word can be adjusted accordingly to use the adjusted sixth task prompt word to guide the fourth model to identify the target entity features of the input image. The implementation process is not described in detail in this application. Of course, if a new target entity feature is determined, it can be added to the sixth task prompt word in the format described above to improve the reliability of the first recognition result.

[0163] In addition, it should be understood that for target entity features of different entities, a corresponding sixth task prompt word can also be configured for each entity according to the example of the sixth task prompt word above. In this case, after the image acquisition device obtains the first image currently taken, it can use the image recognition capability of the image acquisition device itself to identify which entity image the first image is, read the corresponding sixth task prompt word, and then input it and the first image into the fourth model to output the first recognition result of the first image. Of course, in order to reduce the resource consumption of the image acquisition device, or the image acquisition device does not have image recognition capability, the method described in the above example will be adopted. During the process of target entity feature recognition of the first image by the fourth model, each target entity feature listed in the sixth task prompt word will be traversed to output the corresponding first recognition result.

[0164] Step S63, determining the acquisition time corresponding to the first image and the second image in the existing image group;

[0165] Step S64, determining a reference second image in a reference image group of the first image from the second images included in the existing image group according to the acquisition time and the first recognition result;

[0166] In practical applications, when the images to be processed are images captured by an image acquisition device, the images to be processed are usually images captured continuously at a certain time and may have the same subject. For example, the images captured when visiting a car show are usually images of brand A cars captured when visiting brand A cars. The first images to be processed captured during this period are all different images of brand A cars (the same entity); after visiting brand A cars, the images of brand B cars are captured when visiting brand B cars. The first images to be processed captured during this period are all different images of brand B cars (the same entity). Similarly, when taking images while attending lectures, taking images while traveling, taking images while participating in activities, or taking images of a sports event, multiple images of the same entity are usually taken continuously.

[0167] According to the above situation, it can be known that each image that is consecutive in acquisition time is usually a different image of the same entity. Through the contextual entity perception algorithm, combined with the context of the first image currently being processed, it is possible to quickly and accurately determine which entity the first image is an image of, that is, to determine that the first image belongs to the image group corresponding to the entity. Therefore, the present application can determine the acquisition time of the first image, as well as the acquisition time corresponding to each second image in the existing image group, and determine the context of the first image through the acquisition time sequence, that is, the second image that is continuously acquired adjacent to the acquisition time of the first image, so as to subsequently perform contextual entity perception on the first image and determine the target image group to which the first image belongs.

[0168] Based on the above analysis, when the above first recognition result indicates that the first image contains any target entity feature, it is highly likely that the first image is an image of the entity corresponding to the target entity feature and belongs to the image group corresponding to the entity. Compared with directly determining the image group as the target image group to which the first image group belongs, the present application will further select an image group closest to the acquisition time of the first image, that is, an image group with the smallest time difference between the acquisition time and the acquisition time of the first image (that is, the acquisition time is closest to the acquisition time of the first image) from the existing image groups (if the first image is the currently taken image, the acquisition time of each second image in the image group is earlier than the acquisition time of the first image), and determine it as the reference image group of the first image. The entity name of the reference image group can be recorded.

[0169] Afterwards, at least one second image containing the target entity feature (the image containing the target entity feature may be referred to as a high-confidence image) may be selected from each second image included in the reference image group as the reference second image of the first image. In order to conveniently select a high-confidence image in an image group, each image therein may be used as the first image to be processed currently acquired, and in the process of executing the image description generation method proposed in the present application, after determining whether the image contains the target entity feature through the method described in step S61 and step S62, the image is configured with a corresponding recognition result identifier, or after predetermining different identifiers corresponding to different target entity feature recognition results, the image is configured with a first identifier or a second identifier corresponding to the first recognition result of the image, such as configuring the first identifier for the image containing the target entity feature and configuring the second identifier for the image not containing the target entity feature, etc., to determine at least one reference second image in the reference image group that contains the first identifier and whose acquisition time is closest to the acquisition time of the first image, but is not limited to the reference second image determination method described in this embodiment.

[0170] Step S65, comparing the first image with the reference second image for similarity to obtain a second recognition result of the first image;

[0171] Step S66, determining the target image group to which the first image belongs according to the second recognition result;

[0172] Since the reference second image is a correctly grouped historical image and is the image that most accurately represents the entity corresponding to the reference image group in the corresponding reference image group, that is, a high-confidence image, when verifying whether the first image is the image corresponding to the identified target entity image, the application will use the reference second image as a standard to calculate the similarity between the first image and the reference second image. If the similarity is greater than or equal to the first similarity threshold (which can be a higher value, such as 85% or 90%, etc., and the application does not limit its value), it is considered that the entities contained in the two images are the same entity, that is, the first image and its reference second image are different images for the same entity, and the reference image group is the target image group to which the first image belongs. On the contrary, if the similarity is less than the first similarity threshold, it is considered that the entities contained in the first image and its reference second image are not the same entity. For example, if the first image is an image of a brand A car and the reference second image is an image of a brand B car, the first image can be temporarily cached, and the target image group to which the cached first image belongs can be determined after the next high-confidence image to be processed is obtained. The implementation process can refer to the description of the corresponding part below.

[0173] It can be seen that the second recognition result of the first image in the embodiment of the present application indicates whether the first image and the corresponding reference second image are different images collected for the same entity. The second recognition result may include the indication content, or the similarity between the first image and the corresponding reference second image, or the comparison result of the similarity with the first similarity threshold, etc. The present application does not limit the content of the second recognition result. Among them, in the case where the first image and the reference second image are different images collected for the same entity, the entity name (or other image group identifier) ​​of the corresponding reference image group (i.e., the target image group) may be included, so that in the subsequent image description enhancement process, the target image group can be queried based on the entity name.

[0174] In some embodiments, during the implementation of step S66, if the second recognition result indicates that the first image and any reference second image (i.e., the first reference image) are different images collected for the same entity, the reference image group (i.e., the first reference image group) to which the reference second image belongs is determined as the target image group to which the first image belongs. If the second recognition result indicates that the first image and each reference second image (i.e., the first reference image) are not different images collected for the same entity, a new image group is formed by the first images; the new image group is determined as the target entity image group to which the first image belongs.

[0175] Among them, when the target image group only includes the first image, the first image can be directly analyzed according to the method described above to obtain the target image description information of the first image. As time goes by, after adding new images to the target image group, such as a third image whose acquisition time is later than that of the first image, the target image description information of the first image can still be enhanced according to the image description generation method proposed in this application, and the target image description information of the first image obtained can be used as the initial image description information to obtain new target image description information. The implementation process is not described in detail in this application.

[0176] Step S67, caching the first image;

[0177] Step S68, when determining the image group to which the acquired third image belongs, obtaining the similarities between the cached first image and the fourth image and the third image respectively; the third image is an image obtained for the first time after the first image is obtained and contains target entity features corresponding to any entity; the fourth image is at least one image acquired earlier than the first image;

[0178] Step S69: determine whether the similarity reaches a similarity threshold, and determine the image group to which the third image belongs as the target image group of the first image.

[0179] In the embodiment of the present application, according to the method described above, it is determined that the first image does not contain any target entity feature. At this time, the confidence level of the entity (which can be represented by an entity name or other unique identifier) ​​contained in the first image is low based on the entity feature directly recognized from the first image, and the result of determining the image group corresponding to the entity as the target image group is not accurate, which may affect the subsequent enhancement effect of the image description information of the first image. Such first images can be cached in a storage device to determine the target image group to which the cached first image belongs in the subsequent processing process.

[0180] After caching the first image, you can continue to acquire the next image to be processed, which is recorded as the third image. After determining the target image group to which the third image belongs according to the method described in the above embodiment, since the image acquisition process usually involves continuous shooting of the same entity, the entity in the cached first image is likely to be the same entity as the entity in the third image, that is, the entity has not changed between shooting the first image and the third image, but this does not rule out the possibility that the first image and the third image contain different entities.

[0181] Therefore, the present application reads the cached first image, and at least one fourth image whose acquisition time is earlier than the first image (i.e., an image determined in the manner described above that does not contain the target entity feature, and whose acquisition time is continuous with the acquisition time of the first image, usually different images acquired for the same entity), and compares the similarity of each cached image with the third image. If the similarity of a fourth image reaches a similarity threshold (such as the first similarity threshold mentioned above), it is considered that the entity contained in the fourth image and the third image is the same, and each image cached after the fourth image (whose acquisition time is later than the acquisition time of the fourth image) can be assigned to the image group where the third image is located. Therefore, the cached first image and the third image belong to the same image group. Of course, if the similarity of the first image and the third image reaches the similarity threshold, the image group to which the third image belongs is determined as the target image group to which the first image belongs. At this time, if a fifth image whose acquisition time is later than the acquisition time of the first image is also cached, the fifth image can also be assigned to the image group to which the third image belongs, that is, the target image group to which the fifth image belongs is determined.

[0182] Among them, in the above-mentioned similarity comparison process, the similarities between each cached image whose acquisition time is earlier than the acquisition time of the first image and the first image and the third image respectively do not reach the similarity threshold, and these images can continue to be cached, and after waiting for the image group to which the next image to be processed belongs to be determined, continue to determine the target image group to which these cached images belong according to the method described above. It is also possible to select a high-confidence image as a reference image from the previous image group whose acquisition time is later than the acquisition time of the cached image, and determine whether the cached image belongs to the previous image group after similarity comparison. After the various processes described above, the target image group to which each cached image belongs is usually determined, and its image description information can continue to be enhanced.

[0183] In some other embodiments proposed in the present application, if the above-mentioned first recognition result indicates that the first image does not contain the target entity features corresponding to any entity, considering that in the image capturing scenario as shown in the above example, during the image capturing of entity A, the image may be captured by entity B, and then the image may be captured again for entity A, resulting in the multiple images acquired continuously not necessarily being images captured for the same entity.

[0184] Therefore, before caching the first image in the above embodiment, the present application can also obtain two reference image groups that are adjacent to the acquisition time of the first image (if the reference image group of step S64 is recorded as the first reference image group, the reference image group here can be recorded as the second reference image group). Afterwards, considering that the cached first image does not contain the target entity features, and contains very few entity features, or even an entity scene image, directly using the high-confidence image of the entity for similarity calculation, the obtained similarity is low, and it is impossible to determine the target image group to which the first image belongs. Therefore, the present application will select from each second reference image group at least two second images (i.e., high-confidence images) that are adjacent to the first image acquisition time and contain target entity features of any entity, and at least one second image (i.e., low-confidence image, which may be the image closest to the first image acquisition time) that does not contain the target entity features of any entity, and determine them as reference second images (if the reference second image of step S64 is recorded as the first reference image, the reference second image here may be recorded as the second reference image). Afterwards, the first image may be compared with each reference second image in the same reference image group (i.e., each second reference image) to obtain a corresponding similarity comparison result, and the target image group to which the first image belongs may be determined based on the similarity comparison result.

[0185] Among them, for each second reference image obtained from each second reference image group, the similarity can be compared with the first image as a whole. If the obtained similarity reaches the first similarity threshold, the second reference image group can be determined as the target image group to which the first image belongs; otherwise, the target image group to which the first image belongs cannot be determined in this way, and the first image is cached, and the target image group to which the first image belongs is further determined according to the method described in the above embodiment. Of course, for each second reference image obtained from each second reference image group, the similarity is continued to be compared with the first image, and when each corresponding similarity reaches the first similarity threshold, the second reference image group can be determined as the target image group to which the first image belongs; if at least one corresponding similarity does not reach the first similarity threshold, the first image is cached.

[0186] Optionally, in the method described above, before comparing the second reference image with the first image for similarity, it is possible to detect whether the number of images contained in each second reference image group is greater than 3. If so, continue to determine the target image group to which the first image belongs according to the method described above; if not, the similarity between the first image and each image in the two second reference image groups can be calculated to determine the target image group to which the first image belongs. The implementation process is not described in detail in this application.

[0187] Based on the implementation methods of the target image group to which the first image belongs described in the above embodiments, the present application can also use this method to quickly and accurately group a large number of captured images according to entity fine granularity. Figure 7 The image grouping diagram shown in the figure can be sorted according to the acquisition time of each image so as to realize context entity aware grouping later. Afterwards, the target model can be used to identify the target entity features of each image according to the acquisition time sequence, and the image containing any target entity feature can be determined as a high confidence image, such as Figure 7 The images of the corresponding markings of ①, ② and ③ can be realized by referring to the description of the corresponding parts of the above embodiment, which will not be described in detail in this embodiment.

[0188] Afterwards, each high-confidence image can be compared for similarity, and the high-confidence images whose similarity reaches the first similarity threshold are determined as an image group, and the single high-confidence image for which no other high-confidence images whose similarity reaches the first similarity threshold are found is determined as an image group. Of course, in combination with the method for determining the target image group to which the first image belongs described in the above embodiment, the first images to be processed can also be determined in sequence according to the acquisition time sequence, and then the target image group to which they belong can be determined in a corresponding manner by identifying whether they contain target entity features. Combined with the description of the corresponding part of the above embodiment, for the first image that does not contain the target entity feature, that is, the low-confidence image, such as Figure 7As shown, the high-confidence image and adjacent image (i.e., low-confidence image) found from its second reference image group are respectively compared with the first image for similarity to determine the target image group to which the first image belongs. The implementation process is not described in detail in this application.

[0189] It can be seen that the present application realizes image context entity perception according to the image acquisition time sequence and the target entity feature recognition result, improves the accuracy and flexibility of image grouping, and can reliably and accurately determine the target image group to which the first image to be processed belongs, so as to Figure 8 The image description generating method shown obtains target image description information of the first image.

[0190] In practical applications, such as Figure 8 As shown, the first image to be processed may be the currently captured image. In this case, the target image group to which the first image belongs may be determined through context entity perception according to the method described above. The first image to be processed may also be an image obtained from any image group after image grouping is completed. In this case, the image group may be directly determined as the target image group. This application does not limit the source of the first image.

[0191] In order to enhance the initial image description information of the first image, such as Figure 8 As shown, the initial image description information can be enhanced with the help of the image group description information of the target image group and the relevant reference knowledge retrieved based on RAG, so that the obtained target image description information contains more semantic information and contextual background information of the first image. After determining the target image description information of each image, these images and their target image description information can be better organized in combination with the image description information to establish a multimodal information relationship, which can be better associated with the existing knowledge in the knowledge base and can better meet the requirements of downstream tasks such as cross-modal understanding / search, visual question answering, and image-text matching. In particular, in multimodal tasks that do not directly use images but use image description information of images, the image description method proposed in this application is used to obtain more detailed and comprehensive target image description information for each image, thereby improving the performance of multimodal tasks.

[0192] A method for generating an image description provided in an embodiment of the present application is introduced above. A device for executing the above-mentioned method for generating an image description will be introduced below.

[0193] Reference Fig. 9 , is a schematic diagram of the structure of an image description generating device provided in an embodiment of the present application. Fig. 9 As shown, the image description generating device comprises:

[0194] A first image acquisition module 91, used to acquire a first image to be processed;

[0195] a target image group determination module 92, configured to compare and analyze the first image with an existing image group to determine a target image group to which the first image belongs; a second image included in the target image group and the first image are different images collected for the same entity;

[0196] A description information acquisition module 93, used to acquire initial image description information of the first image and current image group description information of the target image group;

[0197] The image description information enhancement module 94 is used to enhance the initial image description information according to the current image group description information to obtain the target image description information of the first image.

[0198] In a possible implementation, the image description information enhancement module 94 may include:

[0199] A first screening unit, configured to screen first reference knowledge related to the current image group description information from a knowledge base according to the current image group description information;

[0200] A first task prompt information obtaining unit, configured to obtain corresponding first task prompt information by using the current image group description information, the first reference knowledge, the first image, the initial image description information and the first task prompt word;

[0201] A first enhancement processing unit, configured to input the first task prompt information into a first model to obtain target image description information of the first image fed back by the first model;

[0202] The first task prompt word is used to guide the first model to compare and analyze the first image and the initial image description information based on the current image group description information and the first reference knowledge to generate the target image description information.

[0203] In a possible implementation, the image description information enhancement module 94 may include:

[0204] A second task prompt information obtaining unit, configured to obtain corresponding second task prompt information by using the target image group, the current image group description information, the first image, the initial image description information and the second task prompt word;

[0205] A second enhancement processing unit, configured to input the second task prompt information into the first model, and obtain new image description information of the first image fed back by the first model, and new image group description information of the target image group;

[0206] A second screening unit, configured to screen second reference knowledge related to the new image group description information from a knowledge base according to the new image group description information;

[0207] A third task prompt information obtaining unit, configured to obtain corresponding third task prompt information by using the second reference knowledge, the first image, the new image description information and the third task prompt word;

[0208] A third enhancement processing unit, configured to input the third task prompt information into the first model to obtain target image description information of the first image fed back by the first model;

[0209] The second task prompt is used to guide the first model to compare and analyze the first image and the initial image description information according to the current image group description information, and after generating the new image description information, to compare and analyze the target image group and the current image group description information according to the new image description information to generate the new image group description information;

[0210] The third task prompt word is used to guide the first model to compare and analyze the first image and the new image description information based on the second reference knowledge to generate the target image description information.

[0211] In a possible implementation, the image description information enhancement module 94 may include:

[0212] A fourth task prompt information obtaining unit, configured to obtain corresponding fourth task prompt information by using the current image group description information, the first image, the initial image description information and a fourth task prompt word;

[0213] a fourth enhancement processing unit, configured to input the fourth task prompt information into the second model to obtain target image description information of the first image fed back by the second model;

[0214] The fourth task prompt word is used to guide the second model to compare and analyze the first image and the initial image description information according to the current image group description information to generate the target image description information.

[0215] In some embodiments, the target image group determination module 92 may include:

[0216] A first recognition unit, used to perform target entity feature recognition on the first image to obtain a first recognition result;

[0217] an acquisition time determining unit, configured to determine the acquisition time corresponding to each of the first image and the second image in the existing image group if the first recognition result indicates that the first image contains a target entity feature corresponding to any entity;

[0218] A first determining unit, configured to determine, according to the acquisition time and the first recognition result, a reference second image in a reference image group of the first image from second images included in an existing image group;

[0219] A first similarity comparison unit is used to compare the first image with the reference second image to obtain a second recognition result of the first image; the second recognition result indicates whether the first image and the corresponding reference second image are different images collected for the same entity;

[0220] The second determining unit is used to determine the target image group to which the first image belongs according to the second recognition result.

[0221] In some embodiments, the target image group determination module 92 may further include:

[0222] a cache unit, configured to cache the first image if the first recognition result indicates that the first image does not contain a target entity feature corresponding to any entity;

[0223] A first acquisition unit, configured to acquire similarities between the first image and the fourth image in the cache and the third image respectively, when determining the image group to which the acquired third image belongs;

[0224] a third determining unit, configured to determine that the similarity reaches a similarity threshold, and determine the image group to which the third image belongs as a target image group for the first image;

[0225] The third image is an image obtained for the first time after the first image is obtained, and contains target entity features corresponding to any entity; the fourth image is at least one image acquired earlier than the first image.

[0226] In some embodiments, the target image group determination module 92 may further include:

[0227] A second acquisition unit is configured to acquire two reference image groups that are adjacent to the acquisition time of the first image if the first recognition result indicates that the first image does not contain a target entity feature corresponding to any entity;

[0228] A first selection unit is used to select, from each of the reference image groups, at least two second images that are adjacent to the first image in acquisition time and contain target entity features of any entity, and at least one second image that does not contain the target entity features of any entity, and determine them as reference second images;

[0229] A second similarity comparison unit, configured to perform a similarity comparison between the first image and each of the reference second images in the same reference image group to obtain a corresponding similarity comparison result;

[0230] A fourth determining unit is used to determine the target image group to which the first image belongs according to the similarity comparison result.

[0231] In a possible implementation, the second determining unit may include:

[0232] a fifth determining unit, configured to determine, if the second recognition result indicates that the first image and any of the reference second images are different images collected for the same entity, a reference image group in which the reference second image belongs as a target image group for the first image;

[0233] A sixth determining unit, configured to, if the second recognition result indicates that the first image and each of the reference second images are not different images collected for the same entity, form a new image group with the first images; and determine the new image group as a target image group to which the first image belongs;

[0234] Wherein, when the target image group only includes the first image, the first image is analyzed to obtain the target image description information of the first image.

[0235] In some embodiments, the description information acquisition module 93 may include:

[0236] A fifth task prompt information obtaining unit, used to obtain corresponding fifth task prompt information using the first image and the fifth task prompt word;

[0237] A first image description generating unit, configured to input the fifth task prompt information into a third model, and obtain initial image description information of the first image fed back by the third model;

[0238] a sixth task prompt information obtaining unit, configured to obtain corresponding sixth task prompt information by using the target image group, image description information of each image included in the target image group, and a sixth task prompt word;

[0239] a fifth enhancement processing unit, configured to input the sixth task prompt information into the third model to obtain current image description information of the target image group fed back by the third model;

[0240] Among them, the fifth task prompt word is used to guide the third model to analyze the first image and generate the initial image description information; the sixth task prompt word is used to guide the third model to analyze each image contained in the target image group and the image description information of the image, and generate the current image description information of the target image group.

[0241] The description information acquisition module 93 may include:

[0242] A reading unit is used to read the stored current image group description information of the target image group; wherein the read current image group description information is generated before acquiring the first image based on the image description information of the second image or third reference knowledge related to the target image group.

[0243] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the image description generation methods provided in the embodiments of the present application.

[0244] Wherein, when loading and executing computer-readable instructions on a computer, all or part of the process or function described in the embodiment of the present application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer-readable instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer-readable instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center to meet the data / instruction transmission requirements in the execution process of the image description generation method proposed in the embodiment of the present application.

[0245] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any image description generation method provided in the embodiment of the present application.

[0246] The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0247] Reference Fig.10 , is a hardware structure diagram of an electronic device applicable to the image description generation method proposed in this application, the electronic device may include but is not limited to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, netbooks, robots or business terminals and other terminal devices, and may also be configured with a server to implement the image description generation method proposed in this application. Fig.10 As shown, the electronic device may include but is not limited to: at least one communication element 101, at least one memory 102 and at least one processor 103, wherein:

[0248] At least one communication element 101, at least one memory 102 and at least one processor 103 can communicate with each other via a bus. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one bidirectional line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0249] The communication element 101 can be used to receive a first image to be processed, such as receiving an image sent by an image acquisition device, or reading a first image to be processed from a database, receiving current image group description information of a target image group, screening relevant reference knowledge from a knowledge base, etc. It can also be used to implement data or instruction transmission between internal components of an electronic device.

[0250] Based on this, in the embodiment of the present application, the communication element 101 may include a communication element corresponding to wireless communication methods such as wifi, Bluetooth, 5G / 6G and other mobile communications, so that the electronic device can realize data transmission with other devices through the communication element, and may also include one or more interfaces supporting wired communication methods, such as general-purpose input / output (GPIO) interface, USB interface, universal asynchronous receiver / transmitter (UART) interface, etc., to realize data transmission between various components inside the electronic device. The present application does not limit the composition structure of the communication element 101 to realize this function and its corresponding communication transmission mechanism, which can be determined according to the circumstances.

[0251] The memory 102 can be used to store multiple computer instructions for implementing the image description generation method proposed in the embodiment of the present application; the processor 103 can load and execute the multiple computer instructions stored in the memory 102 to implement the various steps of the image description generation method proposed in the embodiment of the present application. The implementation process can refer to the description of the corresponding part of the method embodiment above.

[0252] In the embodiment of the present application, the memory 102 may include storage media such as a floppy disk, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. The processor 101 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0253] It should be understood that Fig.10 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiment of the present application. In practical applications, the electronic device may include Fig.10 More or fewer components as shown, or a combination of certain components, such as when the electronic device is a terminal device, may also include microphones, speakers, displays, various sensors, antennas, power modules, RF components, external ports and other input / output components, etc., which can be determined based on processing function requirements, and this application does not give detailed examples one by one.

[0254] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0255] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. In this specification, each embodiment is described in a progressive or parallel manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device and electronic device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

Claims

1. A method for generating an image description, the method comprising: Acquire a first image to be processed; Comparing and analyzing the first image with an existing image group to determine a target image group to which the first image belongs; The second image included in the target image group and the first image are different images collected for the same entity; Acquire initial image description information of the first image and current image group description information of the target image group; The initial image description information is enhanced according to the current image group description information to obtain target image description information of the first image.

2. The method according to claim 1, wherein the step of enhancing the initial image description information based on the current image group description information to obtain the target image description information of the first image comprises: According to the current image group description information, selecting first reference knowledge related to the current image group description information from a knowledge base; Obtain corresponding first task prompt information by using the current image group description information, the first reference knowledge, the first image, the initial image description information and the first task prompt word; Inputting the first task prompt information into a first model to obtain target image description information of the first image fed back by the first model; The first task prompt word is used to guide the first model to compare and analyze the first image and the initial image description information based on the current image group description information and the first reference knowledge to generate the target image description information.

3. The method according to claim 1, wherein the step of enhancing the initial image description information based on the current image group description information to obtain the target image description information of the first image comprises: Obtain corresponding second task prompt information using the target image group, the current image group description information, the first image, the initial image description information, and the second task prompt word; Inputting the second task prompt information into the first model to obtain new image description information of the first image fed back by the first model and new image group description information of the target image group; According to the new image group description information, selecting second reference knowledge related to the new image group description information from a knowledge base; Obtain corresponding third task prompt information by using the second reference knowledge, the first image, the new image description information and the third task prompt word; Inputting the third task prompt information into the first model to obtain target image description information of the first image fed back by the first model; The second task prompt is used to guide the first model to compare and analyze the first image and the initial image description information according to the current image group description information, and after generating the new image description information, to compare and analyze the target image group and the current image group description information according to the new image description information to generate the new image group description information; The third task prompt word is used to guide the first model to compare and analyze the first image and the new image description information based on the second reference knowledge to generate the target image description information.

4. The method according to claim 1, wherein the step of enhancing the initial image description information based on the current image group description information to obtain the target image description information of the first image comprises: Obtain corresponding fourth task prompt information using the current image group description information, the first image, the initial image description information, and a fourth task prompt word; Inputting the fourth task prompt information into the second model to obtain target image description information of the first image fed back by the second model; The fourth task prompt word is used to guide the second model to compare and analyze the first image and the initial image description information according to the current image group description information to generate the target image description information.

5. The method according to claim 1, wherein the step of comparing and analyzing the first image with an existing image group to determine the target image group to which the first image belongs comprises: Performing target entity feature recognition on the first image to obtain a first recognition result; If the first recognition result indicates that the first image contains a target entity feature corresponding to any entity, determining the acquisition time corresponding to each of the first image and the second image in the existing image group; Determine, according to the acquisition time and the first recognition result, a reference second image in a reference image group of the first image from second images included in an existing image group; Comparing the first image with the reference second image for similarity to obtain a second recognition result of the first image; the second recognition result indicates whether the first image and the corresponding reference second image are different images collected for the same entity; A target image group to which the first image belongs is determined according to the second recognition result.

6. The method according to claim 5, wherein the step of comparing and analyzing the first image with an existing image group to determine the target image group to which the first image belongs further comprises: If the first recognition result indicates that the first image does not contain a target entity feature corresponding to any entity, cache the first image; When determining the image group to which the acquired third image belongs, acquiring similarities between the cached first image and fourth image and the third image respectively; determining that the similarity reaches a similarity threshold, and determining the image group to which the third image belongs as the target image group of the first image; The third image is an image obtained for the first time after the first image is obtained, and contains target entity features corresponding to any entity; The fourth image is at least one image acquired earlier than the first image.

7. The method according to claim 5, wherein the step of comparing and analyzing the first image with an existing image group to determine a target image group to which the first image belongs further comprises: If the first recognition result indicates that the first image does not contain a target entity feature corresponding to any entity, obtaining two reference image groups that are adjacent to the acquisition time of the first image; From each of the reference image groups, at least two second images that are adjacent to the first image in acquisition time and contain target entity features of any entity, and at least one second image that does not contain the target entity features of any entity are selected as reference second images; performing a similarity comparison between the first image and each of the reference second images in the same reference image group to obtain a corresponding similarity comparison result; The target image group to which the first image belongs is determined according to the similarity comparison result.

8. The method according to claim 5, wherein determining the target image group to which the first image belongs based on the second recognition result comprises any one of the following: If the second recognition result indicates that the first image and any of the reference second images are different images collected for the same entity, determining the reference image group where the reference second image belongs as the target image group of the first image; If the second recognition result indicates that the first image and each of the reference second images are not different images collected for the same entity, the first images constitute a new image group; the new image group is determined as the target image group to which the first image belongs; in, When the target image group only includes the first image, the first image is analyzed to obtain target image description information of the first image.

9. The method according to any one of claims 1 to 8, wherein the obtaining of initial image description information of the first image and current image group description information of the target image group comprises: Using the first image and the fifth task prompt word, obtaining corresponding fifth task prompt information; Inputting the fifth task prompt information into the third model to obtain initial image description information of the first image fed back by the third model; Using the target image group, image description information of each image included in the target image group and the sixth task prompt word, corresponding sixth task prompt information is obtained, and the sixth task prompt information is input into the third model to obtain current image description information of the target image group fed back by the third model; or reading the stored current image group description information of the target image group; The fifth task prompt word is used to guide the third model to analyze the first image and generate the initial image description information; The sixth task prompt word is used to guide the third model to analyze each image included in the target image group and the image description information of the image, and generate current image description information of the target image group; The read current image group description information is generated before acquiring the first image based on the image description information of the second image or third reference knowledge related to the target image group.

10. An electronic device, comprising: at least one communication element, at least one memory, and at least one processor, wherein: The communication element is used to receive a first image to be processed; The memory is used to store a plurality of computer instructions; The processor is used to load and execute the multiple computer instructions to implement the following steps: acquiring the first image; Comparing and analyzing the first image with an existing image group, determining a target image group to which the first image belongs; a second image included in the target image group and the first image are different images collected for the same entity; Acquire initial image description information of the first image and current image group description information of the target image group; The initial image description information is enhanced according to the current image group description information to obtain target image description information of the first image.

Images