Image description method, device and equipment and computer readable storage medium

Abstract

The invention discloses an image description method, device and equipment and a computer readable storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: describing original image data based on a preset to-be-described feature, and obtaining an initial description text, the initial description text represents description information corresponding to features to be described in the original image data; semantic enhancement processing is conducted on the initial description text from preset five-sense dimensions, an enhanced description text is obtained, and the five-sense dimensions comprise the visual dimension, the tactile dimension, the auditory dimension, the taste dimension and the olfactory dimension; and generating comprehensive description information of the original image data based on the initial description text and the enhanced description text. According to the method and the device, the performance of the multi-modal large model in the cross-modal generation task can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image description method, device, equipment and computer-readable storage medium. Background Art

[0002] With the rapid advancement of information technology and the deepening of global digital transformation, artificial intelligence technology has become a key force in promoting social and economic development. In this context, multi-stable large models have shown great potential in the fields of image and text processing, while also placing higher requirements on the quality of training data.

[0003] At present, the common image data annotation method is mainly structured annotation, which associates each pixel in the image data with label data to provide pixel-level semantic information. However, this type of label data is limited to simple text descriptions of each object in the image data, such as "car", "person" and "tree". Although the annotated image data can be used for simple classification tasks, it has limited recognition effects on complex scenes and detailed features. What is particularly critical is that for multi-stable large models that aim to achieve the functions of image-to-text and text-to-image, single-dimensional description data cannot provide sufficient training support, thereby limiting the performance of large models in cross-modal generation tasks.

[0004] Therefore, how to improve the performance of large multimodal models in cross-modal generation tasks is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide an image description method, apparatus, device and computer-readable storage medium, aiming to improve the performance of large multimodal models in cross-modal generation tasks.

[0006] To achieve the above object, the present application provides an image description method, the image description method comprising:

[0007] Describing the original image data based on the preset features to be described to obtain an initial description text, wherein the initial description text represents the description information corresponding to the features to be described in the original image data;

[0008] Performing semantic enhancement processing on the initial description text from the preset five senses dimensions to obtain an enhanced description text, wherein the five senses dimensions include visual dimension, tactile dimension, auditory dimension, taste dimension and olfactory dimension;

[0009] Comprehensive description information of the original image data is generated based on the initial description text and the enhanced description text.

[0010] In one embodiment, the step of describing the original image data based on the preset features to be described to obtain the initial description text includes:

[0011] Generate a basic description instruction based on a preset feature to be described, wherein the basic description instruction is used to instruct the generation of description information corresponding to the feature to be described, and the feature to be described includes image content, image aesthetics, and image shooting angle;

[0012] The original image data and the basic description instructions are input into a large image-generated text model to obtain an initial description text, wherein the large image-generated text model is used to generate a description text based on the image data.

[0013] In one embodiment, the step of performing semantic enhancement processing on the initial description text from the preset five senses dimension to obtain an enhanced description text includes:

[0014] Annotating the description information corresponding to the feature to be described in the initial description text to obtain an annotated initial description text;

[0015] The annotated initial description text is semantically enhanced from the preset five senses dimension to obtain an enhanced description text.

[0016] In one embodiment, the step of performing semantic enhancement processing on the annotated initial description text from the preset five senses dimension to obtain the enhanced description text includes:

[0017] Generate an enhancement description instruction based on a preset five senses dimension, wherein the enhancement description instruction is used to instruct to perform a semantic enhancement operation according to the five senses dimension;

[0018] The annotated initial description text and the enhanced description instruction are input into a large language model to obtain an enhanced description text, wherein the large language model has natural language understanding and natural language generation capabilities.

[0019] In one embodiment, the step of generating comprehensive description information of the original image data based on the initial description text and the enhanced description text includes:

[0020] The initial description text and the enhanced description text are semantically fused to obtain comprehensive description information of the original image data.

[0021] In one embodiment, after the step of generating the comprehensive description information of the original image data based on the initial description text and the enhanced description text, the method further comprises:

[0022] A picture-text alignment dataset is constructed based on the multiple original image data and the comprehensive description information of each of the multiple original image data, so as to train the multimodal large model to be trained based on the picture-text alignment dataset.

[0023] In one embodiment, the step of constructing an image-text alignment dataset based on the multiple original image data and the comprehensive description information corresponding to each of the multiple original image data includes:

[0024] Based on the preset identification numbers corresponding to the plurality of original image data, respectively, a mapping relationship between the plurality of original image data and the respective comprehensive description information of the plurality of original image data is constructed, wherein one mapping relationship represents an association relationship between one original image data and one comprehensive description information;

[0025] Based on each of the mapping relationships, the plurality of original image data and the plurality of comprehensive description information are saved in a preset data set to form an image-text alignment data set.

[0026] In addition, to achieve the above-mentioned purpose, the present application also provides an image description device, the image description device comprising:

[0027] An initial description module, used to describe the original image data based on preset features to be described, and obtain an initial description text, wherein the initial description text represents description information corresponding to the features to be described in the original image data;

[0028] A semantic enhancement module, used for performing semantic enhancement processing on the initial description text from the preset five senses dimension to obtain an enhanced description text, wherein the five senses dimension includes visual dimension, tactile dimension, auditory dimension, taste dimension and olfactory dimension;

[0029] A comprehensive description module is used to generate comprehensive description information of the original image data based on the initial description text and the enhanced description text.

[0030] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a program for implementing the image description method is stored on the computer-readable storage medium. The program for implementing the image description method is executed by a processor to implement the steps of the image description method as described above.

[0031] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned image description method are implemented.

[0032] The present application provides an image description method, which describes original image data based on preset features to be described to obtain an initial description text, wherein the initial description text represents the description information corresponding to the features to be described in the original image data, and then performs semantic enhancement processing on the initial description text from the preset five senses dimensions to obtain an enhanced description text, wherein the five senses dimensions include visual dimension, tactile dimension, auditory dimension, taste dimension and olfactory dimension, and finally, generates comprehensive description information of the original image data based on the initial description text and the enhanced description text.

[0033] In summary, compared with the traditional method of annotating image data with single-dimensional description data and performing model training based on the annotated image data, the present application first performs a basic description of the original image data to obtain an initial description text, and then semantically enhances the initial description text from five sensory dimensions to obtain an enhanced description text, and generates the final comprehensive description information based on the initial description text and the enhanced description text, thereby achieving the diversity of image description dimensions and improving the quality of image data, thereby improving the performance of the multimodal large model after model training based on high-quality image data in cross-modal generation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 This is a flow chart of the first embodiment of the image description method of the present application;

[0037] Figure 2 A schematic diagram of an image description process involved in an embodiment of the image description method of the present application;

[0038] Figure 3 A schematic diagram of a data set construction process involved in an embodiment of the image description method of the present application;

[0039] Figure 4 This is a schematic diagram of the module structure of the image description device of this application;

[0040] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the image description method in the embodiment of the present application.

[0041] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0043] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0044] The main solution of the present application is: to describe the original image data based on preset features to be described to obtain an initial description text, wherein the initial description text represents the description information corresponding to the features to be described in the original image data; to perform semantic enhancement processing on the initial description text from the preset five senses dimensions to obtain an enhanced description text, wherein the five senses dimensions include visual dimension, tactile dimension, auditory dimension, taste dimension and olfactory dimension; and to generate comprehensive description information of the original image data based on the initial description text and the enhanced description text.

[0045] At present, the common image data annotation method is mainly structured annotation, which associates each pixel in the image data with label data to provide pixel-level semantic information. However, this type of label data is limited to simple text descriptions of each object in the image data, such as "car", "person" and "tree". Although the annotated image data can be used for simple classification tasks, it has limited recognition effects on complex scenes and detailed features. What is particularly critical is that for multi-stable large models that aim to achieve the functions of image-to-text and text-to-image, single-dimensional description data cannot provide sufficient training support, thereby limiting the performance of large models in cross-modal generation tasks.

[0046] This application obtains an initial description text by performing a basic description on the original image data, then semantically enhances the initial description text from five sensory dimensions to obtain an enhanced description text, and generates a final comprehensive description information based on the initial description text and the enhanced description text, thereby achieving diversity in image description dimensions and improving the quality of image data, thereby improving the performance of a large multimodal model trained on high-quality image data in cross-modal generation tasks.

[0047] It should be noted that the execution subject of the method in each embodiment of the image description method of the present application may be an image description system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an image description device capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the image description system as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0048] Based on this, this application proposes an image description method of the first embodiment, please refer to Figure 1 , the image description method comprises steps S10 to S30:

[0049] Step S10, describing the original image data based on the preset features to be described to obtain an initial description text, wherein the initial description text represents the description information corresponding to the features to be described in the original image data;

[0050] It should be noted that the feature items to be described in the image data are pre-set, that is, the features to be described. The original image data refers to image data that has not been labeled or described.

[0051] The original image data is described according to the preset features to be described to obtain description text information (hereinafter referred to as initial description text for distinction), wherein it should be understood that the initial description text represents the description information corresponding to the features to be described in the original image data.

[0052] In one feasible implementation, the features to be described include image content. Specifically, the original image data is input into the large graph-based text model, and the image is annotated by aligning the large graph-based text model to generate simple image description information. The generated description language is required to be in Chinese, and at least the entity information in the original image data is required to be clearly described, such as key entities such as people, animals, plants, buildings, vehicles, and trees, that is, the content in the image. It should be noted that the above-mentioned large graph-based text model can be an open source model such as CPM2.5, or a large model such as GPT4. However, considering the high cost of GPT4, an open source model that can be privately deployed is used for large-scale data generation to save costs.

[0053] In another feasible implementation, the step S10 may include:

[0054] Step S101, generating a basic description instruction based on a preset feature to be described, wherein the basic description instruction is used to instruct the generation of description information corresponding to the feature to be described, and the feature to be described includes image content, image aesthetics, and image shooting angle;

[0055] It should be noted that the features to be described include but are not limited to image content, image aesthetics and image shooting angle, wherein image content includes but is not limited to object (i.e., object name), function (i.e., the purpose of a component or the role it plays in the image), shape (i.e., a specific geometric shape, such as organic or abstract), size (e.g., larger than a reference object or smaller than a reference object), color (e.g., solid color or mottled color), design / pattern (e.g., solid, floral or geometric), texture (e.g., smooth, rough, bumpy, shiny or dull), material (e.g., wood, metal, glass or plastic), state (e.g., good, bad, old, new, damaged or worn), opacity (e.g., transparent, Translucent or opaque), orientation (e.g. upright, horizontal, inverted or tilted), position (e.g. foreground, midground or background), relationship to other components (e.g. interaction or relative spatial arrangement), and text written on the object (i.e. where and how it is written, font and its properties, number of lines, e.g. multiple or single); image aesthetics include but are not limited to overall setting, background, style, overall composition (i.e. arrangement of elements and focal points in the image, e.g. symmetrical, balanced or asymmetrical), lighting (e.g. natural light or artificial light), color palette (i.e. colors or interactions of colors), texture (e.g. smooth, rough, shiny or dull), depth of field (i.e. the entire image or only a portion of it). is the focus, what effect, and how this affects the overall composition), subject (i.e. the main subject of the image, other elements, and how the other elements relate to the subject), mood or feeling (i.e. the overall mood or feeling of the image), number of objects (i.e. the number of primary and secondary objects that appear in the image); the image can be shot at eye level (the camera is at the same level as the subject's eyes. This is the most natural and neutral shooting angle), high angle (the camera is above the subject. This angle can make the subject appear smaller, weaker, or less important), low angle (the camera is below the subject, anywhere below the subject, looking up. This angle can make the subject appear larger, stronger, or more important). Sometimes it's even right under the subject's feet), ground position (the camera is at ground level. This angle captures something at ground level in the frame, that is, the feet, or perhaps a character lying on the ground), tilt (the camera is tilted on its axis. This angle can be used to create a sense of uneasiness or disorientation), bird's eye view (the camera is directly above the subject. This angle can be used to show the subject in relationship to its surroundings), worm's eye view (the camera is directly below the subject. This angle can be used to create a sense of awe or wonder) and bird's eye view or overhead view (the camera is above the subject, shooting from directly above, not at any angle. It is usually closer to the subject than a bird's eye view).

[0056] A description instruction is generated based on a preset descriptive feature (hereinafter referred to as a basic description instruction for distinction). The basic description instruction is used to instruct the generation of description information corresponding to the feature to be described. For example, the basic description instruction may be "please use clear and fluent language to describe the feature to be described in the original image data."

[0057] Step S102, inputting the original image data and the basic description instruction into the image-generated text model to obtain an initial description text, wherein the image-generated text model is used to generate the description text based on the image data.

[0058] It should be noted that the image-to-text model used in the embodiments of the present application is a large model that generates descriptive text based on image data.

[0059] The original image data and the basic description instructions are input into the image-generated text model to obtain the initial description text. It should be noted that the image-generated text model will describe the original image data according to the description rules defined in the basic description instructions to obtain the initial description text.

[0060] Step S20, performing semantic enhancement processing on the initial description text from the preset five senses dimensions to obtain an enhanced description text, wherein the five senses dimensions include visual dimension, tactile dimension, auditory dimension, taste dimension and olfactory dimension;

[0061] It should be noted that the five sense dimensions include visual dimension, tactile dimension, auditory dimension, taste dimension and olfactory dimension.

[0062] It should be understood that the large model of image-generated text can be regarded as a science and engineering man. The text description information it generates can basically describe the image content clearly, but it does not have a good foundation in writing, cannot describe the details in the image more specifically, and cannot add more rhetoric or imagination. Therefore, after obtaining the initial description text, the initial description text is further semantically enhanced from the preset five senses dimension to obtain the processed description text (hereinafter referred to as enhanced description text for distinction). That is, the enhanced description text contains the description information on each sensory dimension in the original image data.

[0063] In this embodiment, the step S20 may include:

[0064] Step S201, annotating the description information corresponding to the feature to be described in the initial description text to obtain an annotated initial description text;

[0065] Step S202, semantic enhancement processing is performed on the annotated initial description text from the preset five senses dimension to obtain an enhanced description text.

[0066] The initial description text generated by the image-generated text model is generally around dozens of words. The description information corresponding to the feature to be described will be annotated according to the description rules specified by the feature to be described to obtain the annotated initial description text. Then, the annotated initial description text is semantically enhanced from the preset five senses dimension to obtain the enhanced description text.

[0067] In this embodiment, the step S202 may include:

[0068] Step S2021, generating an enhancement description instruction based on a preset five senses dimension, wherein the enhancement description instruction is used to instruct to perform a semantic enhancement operation according to the five senses dimension;

[0069] Generate description instructions based on the preset five senses dimensions (hereinafter referred to as enhanced description instructions for distinction), where the enhanced description instructions are used to instruct to perform semantic enhancement operations based on the five senses dimensions. For example, the enhanced description instruction may be "please perform semantic enhancement processing on the initial description text from the five perspectives of vision, touch, hearing, taste, and smell."

[0070] Step S2022: input the annotated initial description text and the enhanced description instruction into a large language model to obtain an enhanced description text, wherein the large language model has natural language understanding and natural language generation capabilities.

[0071] It should be noted that in this embodiment, the large language model is a model with natural language understanding and natural language generation capabilities, such as LLM (Large Language Model). The large language model can be an open source model that can be privately deployed, such as Qianwen, Baichuan or ChatGML.

[0072] The annotated initial description text and the enhanced description instruction are input into the large language model together to obtain the enhanced description text. It should be understood that since the large language model has undergone a large amount of text training, it is similar to an excellent liberal arts student. It can use its profound literary accomplishment to expand the text of the science and engineering man into a more specific and vivid semantic description and write an excellent essay. That is, the large language model (LLM) will output an enhanced semantic description information, expanding a boring short text of dozens of words into a description information with five dimensions of vision, touch, hearing, taste and smell. Here, high-quality semantic description data is basically formed. In order to ensure the stability and consistency of content generation, the five senses method instruction is used for semantic enhancement.

[0073] In this way, the embodiment of the present application utilizes LLM technology to provide a deeper level of semantic understanding than traditional methods, thereby generating more accurate and rich image descriptions, and through the application of the five senses method, describing the image content from multiple sensory dimensions, providing unprecedented richness and detail for the image data. The generated description not only contains basic information, but also has creativity and imagination, which is suitable for AIGC tasks that require high creativity. The automated and intelligent labeling process significantly reduces manual intervention and improves production efficiency and processing speed.

[0074] Step S30: generating comprehensive description information of the original image data based on the initial description text and the enhanced description text.

[0075] The initial description text and the enhanced description text are integrated to obtain comprehensive description information of the original image data, wherein the comprehensive description information represents the description information of the features to be described in the original image data and the description information in five sensory dimensions.

[0076] For example, Figure 2 The figure shows a schematic diagram of the image description process. First, basic description instructions are generated based on the preset features to be described. Then, the original image data is described based on the basic description instructions through the image-to-text big model to generate an initial description text. Enhanced description instructions are generated based on the preset five senses dimensions. The initial description text is semantically enhanced based on the enhanced description instructions through the big language model to generate an enhanced description text. Finally, the initial description text and the enhanced description text are integrated to obtain the comprehensive description information of the original image data.

[0077] In this embodiment, the step S30 may include:

[0078] Step S301 , performing semantic fusion processing on the initial description text and the enhanced description text to obtain comprehensive description information of the original image data.

[0079] Using natural language processing technology, the core content of the initial description text and the sensory details of the enhanced description text are integrated to obtain comprehensive description information. In this way, the embodiment of the present application adopts a unique semantic fusion technology to combine the basic annotation information with the five senses description generated by LLM to generate a more comprehensive and accurate image description.

[0080] Exemplarily, the initial description text "A child is playing on the beach" and the enhanced description text "A child is playing on the golden beach, the sound of waves gently lapping the shore, and the warm sunshine shines on the child's face" are preprocessed; the semantic content of the two texts is analyzed using NLP (Natural Language Processing) technology, and sensory and emotional information such as "child", "beach", and "play" are extracted from the initial description text, and sensory and emotional information such as "golden", "sound of waves", and "warm sunshine" are extracted from the enhanced description text; based on the preset fusion strategy, the comprehensive description information "A child is playing on the golden beach, enjoying the warm sunshine and the gentle sound of waves" is generated.

[0081] The embodiment of the present application performs a basic description on the original image data to obtain an initial description text, then performs semantic enhancement on the initial description text from five sensory dimensions to obtain an enhanced description text, and generates final comprehensive description information based on the initial description text and the enhanced description text, thereby achieving diversity in image description dimensions and improving the quality of image data, thereby improving the performance of a large multimodal model trained on high-quality image data in cross-modal generation tasks.

[0082] In addition, the main annotation methods involved in the current image annotation field include: 1. 2D bounding boxes: mark the location of the target object in the image by drawing a rectangular box around it. This method is simple and easy to use, and is often used in target detection tasks, such as identifying vehicles and pedestrians in autonomous driving; 2. 3D bounding boxes / cuboids: similar to 2D bounding boxes, but with added depth information, marking the length, width and approximate depth of the object. Suitable for object recognition in stereo images, such as three-dimensional spatial positioning of vehicles; 3. Polygons: suitable for irregularly shaped target objects, such as faces and buildings. Annotators draw polygons by placing multiple points on the outer edge of the target object to provide more accurate annotations; 4. Lines and Splines: mainly used to train driving systems to identify lanes and their boundaries. Annotators draw boundary lines along established machine learning methods to help autonomous driving systems understand the boundaries they are in; 5. Semantic Segmentation: associate each pixel in the entire image with a label to provide pixel-level semantic information. Commonly used in fields such as autonomous driving and medical image analysis, such as segmenting roads, buildings, cyclists, pedestrians, etc. in traffic images; 6. OCR transcription: Use optical character recognition (OCR) technology to identify and transcribe text content in images to help train and improve image and text recognition models. Usually used in combination with rectangular box annotation to locate and identify text areas in images; 7. Attribute discrimination: Identify target objects in images and annotate their attributes, such as gender, age, expression, etc. in face recognition. This helps the model better understand the image content and improve the accuracy of classification and recognition; 8. Key point annotation: Identify important points of interest in images, such as joints or facial features of the human body. Commonly used in tasks such as posture prediction and face recognition; 9. 2D / 3D fusion annotation: Annotate image data collected by 2D and 3D sensors at the same time and establish associations. Suitable for target detection and tracking in complex scenes, it is necessary to consider both two-dimensional and three-dimensional information; 10. Image classification: Simple text labels for images, such as "car", "person", "tree", etc. It is suitable for simple classification tasks, but has limited recognition effect on complex scenes and detailed features; 11. Combination of deep learning and image annotation: Deep learning technology, especially convolutional neural networks, can automatically extract image features and improve the automation and accuracy of image annotation. The training and optimization of deep learning models are the key to ensuring the accuracy of image annotation.

[0083] The above annotation methods play an important role in different application scenarios, but there are still some defects, such as: the limitations of labeled annotation. Existing technologies usually focus on assigning simple labels or categories to images, which is suitable for basic classification tasks. However, this method lacks descriptive details and cannot capture complex scenes and contextual information in images, which limits its application in more advanced AI tasks; lack of contextual understanding. Labeled annotations usually do not contain spatial relationships or interactions between objects in the image, which may cause the model to be insufficient in understanding the context of the image; lack of semantic information. Simple annotations cannot provide semantic information such as the attributes, states, or actions of objects in the image, which is essential for in-depth understanding and high-level reasoning of images; cannot support AIGC (AI Generated Content) tasks. For AIGC tasks such as text-to-image and image-to-text, richer and more detailed image descriptions are required to generate or understand text and image content. Existing technologies cannot provide sufficient information to support such creative and generative tasks; the lack of description of the five senses, namely vision, hearing, taste, touch and smell, which can provide more comprehensive and vivid semantic information for images; the degree of automation and intelligence is insufficient. Although some automated tools can assist the labeling process, they are usually rule-based and lack the ability of intelligent understanding and creative description.

[0084] Thus, compared with the above-mentioned annotation methods, the technical effects achieved by the embodiments of the present application include: enhancing the semantic richness of image annotation. Through the LLM technology, the problem that the traditional image annotation method only provides basic category labels and lacks deep semantic understanding of the image content is solved; providing multi-dimensional sensory description. Using the five senses method, the image can be described in five dimensions: vision, hearing, taste, touch and smell, which solves the problem that the existing technology cannot fully capture and express the sensory information in the image; improving the creativity and expressiveness of image annotation. The annotation information enhanced by the LLM model can generate more creative and expressive text descriptions, supporting AIGC tasks such as text-to-image and image-to-text; improving the contextual understanding ability of image annotation. Through semantic enhancement technology, the model can better understand the contextual information in the image, including the relationship between objects. and interaction; enhance the automation and intelligence level of image annotation. Compared with traditional rule-based automation tools, the use of advanced LLM technology improves the intelligence of the image annotation process; improve the quality and consistency of image annotation data. Through the consistent output of the LLM model, the subjectivity and inconsistency in the manual annotation process are reduced, and the overall quality of the data is improved; support cross-domain and personalized image annotation needs, with good adaptability and scalability, and can customize image annotation solutions according to different fields or personalized needs; promote the application of image annotation technology in emerging fields, and provide new possibilities for the application of image annotation in emerging fields such as virtual reality, augmented reality, and autonomous driving; reduce the labor cost of image annotation. Using large model technology for image annotation can greatly reduce the labor cost of human annotation and improve annotation efficiency and quality. At the same time, because the project uses open source large models for overall design, it also avoids the high cost of online large model services.

[0085] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. On this basis, after the step S30, the image description method further includes:

[0086] Step A10: constructing a picture-text alignment dataset based on the multiple original image data and the comprehensive description information of each of the multiple original image data, so as to train the multimodal large model to be trained based on the picture-text alignment dataset.

[0087] It should be noted that in the actual image data description process, multiple image data can be processed simultaneously, that is, batch processing, so as to improve the efficiency of image description. Therefore, each original image data is described to obtain the comprehensive description information corresponding to each original image data, and then a picture-text alignment dataset is constructed based on multiple original image data and each comprehensive description information, that is, the picture-text alignment dataset contains multiple sets of one-to-one corresponding image data and description information. In this way, the picture-text alignment dataset is used as a training sample set to train the multimodal large model to be trained, and a higher performance multimodal large model is obtained.

[0088] For example, Figure 3 The figure shows a schematic diagram of the dataset construction process. First, the original image data and the basic description instructions for guiding the basic description of the image are input into the image-to-text large model to obtain the initial description text; then the initial description text and the enhanced description instructions for guiding the semantic enhancement of the five senses are input into the large language model to obtain the enhanced description text; the comprehensive description information is determined based on the initial description text and the enhanced description text; finally, the image-text alignment dataset is constructed based on the original image data and the comprehensive description information.

[0089] In this embodiment, the step A10 may include:

[0090] Step A101, based on the preset identification numbers corresponding to the plurality of original image data, constructing a mapping relationship between the plurality of original image data and the comprehensive description information of the plurality of original image data, wherein one mapping relationship represents an association relationship between one original image data and one comprehensive description information;

[0091] It should be noted that an ID, i.e., an identification number, is assigned to each original image data in advance, the initial description text and enhanced description text generated in the image description process of each original image data are recorded, and the initial description text and enhanced description text are marked based on the identification number of the original image data, so as to construct a mapping relationship between each original image data and its comprehensive description information. Among them, each mapping relationship represents the association relationship between an original image data and a comprehensive description information.

[0092] Step A102: saving the plurality of original image data and the plurality of comprehensive description information into a preset data set based on the respective mapping relationships to form an image-text alignment data set.

[0093] It should be noted that an empty data set is pre-built, that is, the above-mentioned preset data set.

[0094] Based on each mapping relationship, multiple original image data and multiple comprehensive description information are saved in a preset data set to form an image-text alignment data set.

[0095] Exemplarily, the following table 1 shows the structure table of the image-text alignment dataset, which contains records of original image data, initial description text, and enhanced description text. The description of each original image data includes ID, image naming rule, original image address, original image width, and original image height; the description of each initial description text includes text version number, naming rule of initial description text, and initial description text; the description of enhanced description text includes text version number, naming rule of enhanced description text, enhanced description text, text similarity between enhanced description text and initial description text, and text enhancement model parameter version number. In this way, each data can be effectively managed in the image-text alignment dataset.

[0096] Table 1

[0097]

[0098] In this way, the embodiment of the present application performs image description on multiple original image data to obtain the initial description text and enhanced description text corresponding to each original image data, and further obtains the comprehensive description information of each original image data, and then constructs a picture-text alignment dataset based on each original image data and each comprehensive description information, that is, generates a high-quality sample dataset for subsequent training of the model to be trained based on the picture-text alignment dataset, and improves the cross-modal processing capability of the trained model.

[0099] The present application also provides an image description device, please refer to Figure 4 , the image description device comprises:

[0100] An initial description module 10 is used to describe the original image data based on preset features to be described, and obtain an initial description text, wherein the initial description text represents description information corresponding to the features to be described in the original image data;

[0101] The semantic enhancement module 20 is used to perform semantic enhancement processing on the initial description text from the preset five senses dimension to obtain an enhanced description text, wherein the five senses dimension includes a visual dimension, a tactile dimension, an auditory dimension, a taste dimension and an olfactory dimension;

[0102] The comprehensive description module 30 is used to generate comprehensive description information of the original image data based on the initial description text and the enhanced description text.

[0103] Optionally, the initial description module 10 is further used for:

[0104] Generate a basic description instruction based on a preset feature to be described, wherein the basic description instruction is used to instruct the generation of description information corresponding to the feature to be described, and the feature to be described includes image content, image aesthetics, and image shooting angle;

[0105] The original image data and the basic description instructions are input into a large image-generated text model to obtain an initial description text, wherein the large image-generated text model is used to generate a description text based on the image data.

[0106] Optionally, the semantic enhancement module 20 is further used for:

[0107] Annotating the description information corresponding to the feature to be described in the initial description text to obtain an annotated initial description text;

[0108] The annotated initial description text is semantically enhanced from the preset five senses dimension to obtain an enhanced description text.

[0109] Optionally, the semantic enhancement module 20 is further used for:

[0110] Generate an enhancement description instruction based on a preset five senses dimension, wherein the enhancement description instruction is used to instruct to perform a semantic enhancement operation according to the five senses dimension;

[0111] The annotated initial description text and the enhanced description instruction are input into a large language model to obtain an enhanced description text, wherein the large language model has natural language understanding and natural language generation capabilities.

[0112] Optionally, the comprehensive description module 30 is further used for:

[0113] The initial description text and the enhanced description text are semantically fused to obtain comprehensive description information of the original image data.

[0114] Optionally, the image description device further includes a data set construction module, and the data set construction module is used to:

[0115] A picture-text alignment dataset is constructed based on the multiple original image data and the comprehensive description information of each of the multiple original image data, so as to train the multimodal large model to be trained based on the picture-text alignment dataset.

[0116] Optionally, the data set construction module is further used to:

[0117] Based on the preset identification numbers corresponding to the plurality of original image data, respectively, a mapping relationship between the plurality of original image data and the respective comprehensive description information of the plurality of original image data is constructed, wherein one mapping relationship represents an association relationship between one original image data and one comprehensive description information;

[0118] Based on the mapping relationships, the plurality of original image data and the plurality of comprehensive description information are saved in a preset data set to form an image-text alignment data set.

[0119] The image description device provided in the embodiment of the present application adopts the image description method in the above embodiment, which can solve the technical problem of how to improve the performance of a large multimodal model in a cross-modal generation task. Compared with the prior art, the beneficial effects of the image description device provided in the embodiment of the present application are the same as the beneficial effects of the image description method provided in the above embodiment, and the other technical features in the image description device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0120] The present application provides an image description device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image description method in the above-mentioned embodiment 1.

[0121] Reference below Figure 5 , which shows a schematic diagram of the structure of an image description device suitable for implementing the embodiment of the present application. The image description device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The image description device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0122] like Figure 5As shown, the image description device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the image description device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the image description device to communicate with other devices wirelessly or by wire to exchange data. Although the image description device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0123] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0124] The image description device provided by the present application adopts the image description method in the above embodiment, which can solve the technical problem of how to improve the performance of a large multimodal model in a cross-modal generation task. Compared with the prior art, the beneficial effects of the image description device provided by the present application are the same as the beneficial effects of the image description method provided by the above embodiment, and the other technical features in the image description device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0125] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0127] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the image description method in the above-mentioned embodiment.

[0128] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0129] The computer-readable storage medium may be included in the image description device; or may exist independently without being assembled into the image description device.

[0130] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the image description device, the image description device: describes the original image data based on the preset features to be described to obtain an initial description text, wherein the initial description text represents the description information corresponding to the features to be described in the original image data; performs semantic enhancement processing on the initial description text from the preset five senses dimensions to obtain an enhanced description text, wherein the five senses dimensions include visual dimension, tactile dimension, auditory dimension, taste dimension and olfactory dimension; generates comprehensive description information of the original image data based on the initial description text and the enhanced description text.

[0131] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0134] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image description method, and can solve the technical problem of how to improve the performance of multimodal large models in cross-modal generation tasks. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the image description method provided in the above-mentioned embodiment, and will not be repeated here.

[0135] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned image description method when executed by a processor.

[0136] The computer program product provided in this application can improve the performance of a large multimodal model in cross-modal generation tasks. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the image description method provided in the above embodiment, which will not be repeated here.

[0137] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. An image description method, characterized in that: The image description method comprises: Describing the original image data based on the preset features to be described to obtain an initial description text, wherein the initial description text represents the description information corresponding to the features to be described in the original image data; Performing semantic enhancement processing on the initial description text from the preset five senses dimensions to obtain an enhanced description text, wherein the five senses dimensions include visual dimension, tactile dimension, auditory dimension, taste dimension and olfactory dimension; Comprehensive description information of the original image data is generated based on the initial description text and the enhanced description text.

2. The method according to claim 1, characterized in that The step of describing the original image data based on the preset features to be described to obtain an initial description text includes: Generate a basic description instruction based on a preset feature to be described, wherein the basic description instruction is used to instruct the generation of description information corresponding to the feature to be described, and the feature to be described includes image content, image aesthetics, and image shooting angle; The original image data and the basic description instructions are input into a large image-generated text model to obtain an initial description text, wherein the large image-generated text model is used to generate a description text based on the image data.

3. The method according to claim 1, characterized in that The step of performing semantic enhancement processing on the initial description text from the preset five senses dimension to obtain an enhanced description text comprises: Annotating the description information corresponding to the feature to be described in the initial description text to obtain an annotated initial description text; The annotated initial description text is semantically enhanced from the preset five senses dimension to obtain an enhanced description text.

4. The method according to claim 3, characterized in that The step of performing semantic enhancement processing on the annotated initial description text from the preset five senses dimension to obtain an enhanced description text includes: Generate an enhancement description instruction based on a preset five senses dimension, wherein the enhancement description instruction is used to instruct to perform a semantic enhancement operation according to the five senses dimension; The annotated initial description text and the enhanced description instruction are input into a large language model to obtain an enhanced description text, wherein the large language model has natural language understanding and natural language generation capabilities.

5. The method according to claim 1, characterized in that The step of generating comprehensive description information of the original image data based on the initial description text and the enhanced description text comprises: The initial description text and the enhanced description text are semantically fused to obtain comprehensive description information of the original image data.

6. The method according to any one of claims 1 to 5, characterized in that After the step of generating the comprehensive description information of the original image data based on the initial description text and the enhanced description text, the method further comprises: A picture-text alignment dataset is constructed based on the multiple original image data and the comprehensive description information of each of the multiple original image data, so as to train the multimodal large model to be trained based on the picture-text alignment dataset.

7. The method according to claim 6, characterized in that The step of constructing a picture-text alignment data set based on the multiple original image data and the comprehensive description information corresponding to each of the multiple original image data comprises: Based on the preset identification numbers corresponding to the plurality of original image data, respectively, a mapping relationship between the plurality of original image data and the respective comprehensive description information of the plurality of original image data is constructed, wherein one mapping relationship represents an association relationship between one original image data and one comprehensive description information; Based on the mapping relationships, the plurality of original image data and the plurality of comprehensive description information are saved in a preset data set to form an image-text alignment data set.

8. An image description device, characterized in that: The image description device comprises: An initial description module, used to describe the original image data based on preset features to be described, and obtain an initial description text, wherein the initial description text represents description information corresponding to the features to be described in the original image data; A semantic enhancement module, used for performing semantic enhancement processing on the initial description text from the preset five senses dimension to obtain an enhanced description text, wherein the five senses dimension includes visual dimension, tactile dimension, auditory dimension, taste dimension and olfactory dimension; A comprehensive description module is used to generate comprehensive description information of the original image data based on the initial description text and the enhanced description text.

9. An image description device, characterized in that: The image description device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image description method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image description method according to any one of claims 1 to 7 are implemented.

Images