Image description method and device, and model training method and device

Abstract

The application provides an image description method and device, and a training method and device of a model. The image description method comprises the following steps: acquiring a target image; performing feature extraction on the target image by using an image model to obtain pre-description data; identifying first identification data in the pre-description data, and obtaining target data corresponding to the first identification data in a conversion set according to the first identification data; and replacing second identification data in the pre-description data with the target data to obtain description data. The technical solution of the application can improve the generation efficiency and reliability of the description data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an image description method, apparatus, and a training method and apparatus for a model. Background Technology

[0002] Image captioning is an interdisciplinary task combining computer vision and natural language processing. Its purpose is to generate natural language descriptions of given images using computers. This image captioning method has been widely applied with the proficient use of deep learning. However, in this field, image captioning methods typically target images captured under natural light, lacking methods for images captured under non-natural light. Furthermore, due to the significant differences between images captured under natural and non-natural light, cross-modal descriptions are less effective, and manual descriptions suffer from low efficiency and consistency. Summary of the Invention

[0003] In view of this, the embodiments of this application aim to provide an image description method and a model training method to solve at least one technical problem in the prior art or related art.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] In a first aspect, an image description method is provided, comprising: acquiring a target image, the target image being captured under a first type of light source; extracting features from the target image using an image model to obtain pre-description data, the pre-description data including a set of textual description information for each item in the target image; identifying first identification data in the pre-description data; obtaining target data corresponding to the first identification data in the transformation set through a transformation set, the first identification data being material data of each item in the pre-description data; the transformation set including a mapping set between the material data and the pseudo-color data of the image obtained by the material corresponding to the material data under illumination by a second type of light source; the target data being pseudo-color data; and the first type of light source being different from the second type of light source; and replacing the second identification data in the pre-description data with the target data to obtain description data, the second identification data being color data of each item in the pre-description data.

[0006] In some alternative embodiments of this application, the second type of light source includes X-rays.

[0007] In some optional embodiments of this application, the second identification data in the pre-description data is replaced with the target data to obtain the description data, including: when the existence of third identification data is identified in the description data, a first label is added to the third identification data to obtain the description data, wherein the third identification data includes the name data of prohibited items for security inspection, and the first label includes the prohibited item entry marked at the third identification data.

[0008] In some optional embodiments of this application, replacing the second identification data in the pre-description data with the target data to obtain description data includes: replacing the second identification data in the pre-description data with the target data, and adding a second label to the pre-description data to obtain description data, wherein the second label includes type data of the second type of light source.

[0009] In some alternative embodiments of this application, the second label is placed before the descriptive data.

[0010] In some alternative embodiments of this application, the first type of light source is natural light.

[0011] In some alternative embodiments of this application, the image model is one of Qwen2.5-VL, Qwen3-VL, and InternLM-XComposer-2.5.

[0012] In some optional embodiments of this application, replacing the second identification data in the pre-description data with the target data to obtain the description data includes: replacing the second identification data in the pre-description data with the target data and removing the first identification data in the pre-description data to obtain the description data.

[0013] Secondly, a training method for a model is provided, comprising: acquiring a sample training set, which includes sample second-type images and sample description data, wherein the sample second-type images are taken by the object of the target image under a second-type light source, and the target image is taken under a first-type light source, the first-type light source and the second-type light source are different, and the sample description data is description data obtained based on the above image description method; and training the model using the sample training set to obtain a second-type image description model.

[0014] Thirdly, an image description method is provided, comprising: acquiring a target second-type image; inputting the target second-type image into a second-type image description model to generate description data of the target second-type image, wherein the second-type image description model is obtained based on the training method of the aforementioned model.

[0015] Fourthly, an image description device is provided, comprising: an acquisition module for acquiring a target image, the target image being captured under a first type of light source; an extraction module for extracting features from the target image using an image model to obtain pre-description data, the pre-description data including a set of textual description information for each item in the target image; an identification module for identifying first identification data in the pre-description data, and obtaining target data corresponding to the first identification data in the transformation set through a transformation set, the first identification data being material data of each item in the pre-description data, the transformation set including a mapping set between the material data and the image pseudo-color data of the material corresponding to the material data obtained under illumination by a second type of light source, the target data being pseudo-color data, the first type of light source being different from the second type of light source; and a replacement module for replacing second identification data in the pre-description data with the target data to obtain description data, the second identification data being color data of each item in the pre-description data.

[0016] Fifthly, a model training apparatus is provided, comprising: an acquisition module for acquiring a sample training set, the sample training set including sample second-type images and sample description data, wherein the sample second-type images are taken by photographing the object of the target image under a second-type light source, the target image is taken under a first-type light source, the first-type light source and the second-type light source are different, and the sample description data is description data obtained based on the above-mentioned image description method; and a training module for training the model using the sample training set to obtain a second-type image description model.

[0017] In a sixth aspect, an electronic device is provided, comprising: a processor, a memory, and a communication interface; the memory and the communication interface are coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions; wherein, when the processor executes the computer instructions, the electronic device performs the method of the first aspect, the method of the second aspect, or the method of the third aspect described above.

[0018] In a seventh aspect, a computer-readable storage medium is provided, which stores a computer program, the computer program including program instructions, which, when executed by a processor, perform the method of the first aspect, the method of the second aspect, or the method of the third aspect described above.

[0019] Eighthly, an apparatus (e.g., a chip system) is provided, comprising a processor for supporting an electronic device in implementing the methods of the first, second, or third aspects described above.

[0020] In some alternative embodiments of this application, the device further includes a memory for storing program instructions and data necessary for the electronic device. When the device is a chip system, it may be composed of chips or may include chips and other discrete components.

[0021] In this embodiment, a target image captured under a first type of light source can be acquired first. Then, feature extraction is performed on the target image based on an image model to obtain pre-description data. This pre-description data set includes the material data and color data of each item in the target image. The pre-description data is then identified to identify the material data in the pre-description data. Based on the correspondence of the material data in the conversion set, such as the mapping relationship between the material data and the image pseudo-color data of the material corresponding to the material data obtained under the illumination of the second type of light source in the conversion set, the corresponding pseudo-color data is obtained. Then, the color data of each item in the pre-description data is replaced with the pseudo-color data to obtain the description data. Unlike the first type of light source, which has a relevant image description model, the first type of light source can be used to obtain an initial natural language description. Then, based on the obtained natural language description, the parts of the natural language description that differ due to the differences between the first and second type of light sources are filtered out. After passing through a transformation set, the material features in the initial natural language description are used to obtain a partial description of the image captured by the second type of light source. This part is then replaced to obtain the natural language description of the image captured by the second type of light source. This achieves the goal of obtaining the natural language description of the image captured by the second type of light source using the relevant image description model of the first type of light source, avoiding the problems of directly using the model for cross-modal description, as well as the problems of reduced efficiency and consistency caused by using manual description. Attached Figure Description

[0022] Figure 1 The diagram shown is a flowchart illustrating an image description method provided in an exemplary embodiment of this application.

[0023] Figure 2 The diagram shown is a flowchart illustrating a training method for a model provided in an exemplary embodiment of this application.

[0024] Figure 3 The diagram shown is a flowchart illustrating an image description method provided in another exemplary embodiment of this application.

[0025] Figure 4 The diagram shown is a flowchart of an image description apparatus provided in an exemplary embodiment of this application.

[0026] Figure 5 The diagram shown is a flowchart of a training apparatus for a model provided in an exemplary embodiment of this application.

[0027] Figure 6 The diagram shown is a flowchart of an image description apparatus provided in another exemplary embodiment of this application.

[0028] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that “ / ” means “or,” for example, A / B can mean A or B; “and / or” in the text is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can mean: A alone, A and B simultaneously, and B alone.

[0030] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0031] The terms "first" and "second" in the following embodiments of this application are for descriptive purposes only and should not be construed as implying relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0032] Image captioning is an interdisciplinary task combining computer vision and natural language processing. Its purpose is to generate natural language descriptions of given images using computers. This image captioning method has been widely applied with the proficient use of deep learning. However, in this field, image captioning methods typically target images captured under natural light, lacking methods for images captured under non-natural light. Furthermore, due to the significant differences between images captured under natural and non-natural light, cross-modal descriptions are less effective, and manual descriptions suffer from low efficiency and consistency.

[0033] To address this, the application provides an image description method. First, a target image captured under a first type of light source is acquired. Then, features are extracted from the target image based on an image model to obtain pre-description data, which in turn yields a set of textual description information for each item in the target image. This textual description information set includes material data and color data for each item. Further, the pre-description data is identified to determine the material data. Based on the correspondence of material data in a transformation set (e.g., if the transformation set contains a mapping relationship between material data and the corresponding material's image pseudo-color data obtained under a second type of light source), the corresponding pseudo-color data is obtained. Then, the color data of each item in the pre-description data is replaced with the pseudo-color data to obtain the description data. Since the first and second types of light sources differ, an initial natural language description can be obtained based on the first type of light source, which has a relevant image description model. Finally, the natural language descriptions obtained are filtered to remove the portions that differ due to the differences between the first and second types of light sources. After transformation, a partial description of the image captured by the second type of light source is obtained through the material features in the initial natural language description. This partial description is then replaced to obtain the natural language description of the image captured by the second type of light source. This method utilizes the image description model related to the first type of light source to obtain the natural language description of the image captured by the second type of light source, avoiding the problems of directly using the model for cross-modal description and the efficiency and consistency reduction issues caused by using manual description.

[0034] The image description method provided in this application will be described in detail below with reference to the accompanying drawings.

[0035] like Figure 1 As shown, the image description method may specifically include S110-S140.

[0036] Step S110: Acquire the target image, which is captured under the first type of light source.

[0037] A target image to be described is acquired. This target image can be an image in JPG, PNG, or other formats, or it can be an image frame from a video. The first type of light source can be a light source with a corresponding image description model, such as natural light. In this case, the image description model can be Qwen2.5-VL, Qwen3-VL, or InternLM-XComposer-2.5. In this embodiment, natural light is used as the first type of light source, and Qwen2.5-VL is used as the image description model for explanation.

[0038] Step S120: Extract features from the target image using an image model to obtain pre-description data, which includes a set of textual description information for each item in the target image.

[0039] The image model can be an image description model or a multimodal model. The specific choice of image model depends on the first type of light source. For example, when using Qwen2.5-VL as the image model, the feature extraction process can be roughly as follows: use visual encoding to directly process the target image, segment it into a certain number of image blocks, then perform feature compression and alignment, and finally perform multimodal fusion and understanding to obtain image description text, that is, obtain pre-description data.

[0040] Furthermore, prompts can be input during the aforementioned feature extraction process, such as asking to identify all visible objects in the image, and the pre-description data can be output in the following format:

[0041] Item Category:

[0042] Exterior color:

[0043] Material category:

[0044] Estimated dimensions:

[0045] Relative position (left / right / front / back / up / down):

[0046] Furthermore, the pre-description data can specify the material category of each object in its natural environment. For example, taking X-rays as the second type of light source, four categories can be set (organic materials, inorganic materials, mixed materials, and materials impermeable by X-rays), and each item belongs to one of these categories. For instance, mixed materials could be glass, etc.

[0047] It is understandable that image models can also be retrieval models using publicly available benchmark datasets such as COCO Caption and Flickr30k Caption. For example, using COCO Caption as an image description dataset, multiple feature images that match the target image can be retrieved from multiple preset image samples in the dataset, along with the descriptive text corresponding to these multiple feature images, thus obtaining pre-description data.

[0048] In some optional embodiments of this application, when using Qwen2.5-VL, the prompt words also include information that should not be omitted when outputting, that is, if there is uncertainty in the identification process, the item category should be added with a prefix such as "suspected".

[0049] Of course, the above are only limited examples of image models. In other possible implementations of this application, the image model can also be any other image description model, which is not limited here.

[0050] Step S130: Identify the first identification data in the pre-description data, and obtain the target data corresponding to the first identification data in the transformation set through the transformation set. The first identification data is the material data of each item in the pre-description data. The transformation set includes the mapping set between the material data and the image pseudo-color data of the material corresponding to the material under the illumination of the second type of light source. The target data is pseudo-color data. The first type of light source and the second type of light source are different.

[0051] In this embodiment, X-rays are used as an example of the second type of light source. The conversion set can have a mapping relationship between a first set and a second set. The first set can include the material of the object, such as organic materials, inorganic materials, mixed materials, and materials that are impermeable to X-rays. The second set can include the pseudo-color information of organic materials, inorganic materials, mixed materials, and materials that are impermeable to X-rays in the X-ray image, such as orange / light orange, blue, green, and black. It can be understood that the mapping relationship between the first set and the second set can be that organic materials correspond to orange / light orange, inorganic materials correspond to blue, mixed materials correspond to green, and materials that are impermeable to X-rays correspond to black.

[0052] Of course, the above is only one example of the second type of light source being X-ray. In other possible implementations of this application, the conversion set can also be any other set that can be converted for its corresponding second type of light source to obtain target data, and there are no restrictions here.

[0053] For example, the second type of light source can also be infrared light, gamma rays, etc.

[0054] Step S140: Replace the second identification data in the pre-description data with the target data to obtain the description data. The second identification data is the color data of each item in the pre-description data.

[0055] The second identification data can be the color of the corresponding item. For example, if the image contains yellow sneakers and a silver camera, then the second identification data includes yellow and silver. Combining other descriptive information in the image, the yellow sneakers are made of organic materials and the silver camera is made of metallic materials. Therefore, the corresponding target data can be orange pseudo-color sneakers and blue pseudo-color camera.

[0056] In this embodiment, images with relevant image model light source types can be used to describe the same photographed object under the same environment. For example, if an X-ray image description is required, natural light photography can be added before or after the X-ray imaging point, ensuring the natural light image and the X-ray image have the same photographed object and environment. Image model analysis is performed on the obtained natural light image to obtain a natural light description. Based on the different false colors of X-ray images of different materials, the material of each item in the natural light description is analyzed, and different materials and their corresponding false colors are filtered out. Then, the false colors are used to replace the colors of each item in the natural light description to obtain the X-ray image description. In the aforementioned process, the image description originates from a natural light image model. This type of model is relatively mature, has many types, obtains high-precision descriptions, and is conducive to multi-faceted verification. It enables rapid description of X-ray images, improving the low efficiency and inconsistent nature of manual descriptions. Furthermore, the description obtained from the natural light image model used in the description process is still the description corresponding to the natural light image, reducing the difficulty of describing direct cross-modal differences. This improves the efficiency and reliability of generating descriptive data for images obtained under second-type light source illumination.

[0057] In some alternative embodiments of this application, the second type of light source includes X-rays.

[0058] When the second type of light source is X-ray, it can effectively solve the problems of complex X-ray imaging characteristics, false color of objects, and difficulty in directly describing their appearance with image models. It solves the problem of quickly constructing X-ray image description annotation sets and promotes the development of X-ray multimodal models.

[0059] In some optional embodiments of this application, step S140, replacing the second identification data in the pre-description data with the target data to obtain description data, includes: if the existence of third identification data is identified in the description data, adding a first label to the third identification data to obtain description data, wherein the third identification data includes security inspection prohibited item name data, and the first label includes prohibited item entries marked at the third identification data.

[0060] The third identification data can be entries from a prohibited items database. This database can be constructed by collecting entries in advance, or it can be a collection of prohibited items entries extracted from regulations such as the "List of Prohibited Items".

[0061] For example, the first label can be a prefix or suffix of the prohibited item term set in the third identification data. For instance, the descriptive data could be, "A blue pseudo-colored metallic lighter (prohibited item) is visible in the image, surrounded by orange pseudo-colored organic materials, including clothing and footwear." Here, "(prohibited item)" is the first label.

[0062] Understandably, if the third identification data is not present in the identified descriptive data, a third label is added to the descriptive data to obtain new descriptive data. This third label includes a "no prohibited items" flag in the descriptive data. This third label can be a prefix or suffix indicating "no prohibited items" at the beginning or end of the descriptive data. For example, the descriptive data could be: "The image shows a pair of orange pseudo-color sneakers, a blue pseudo-color camera, a green pseudo-color plastic passport case, and an orange pseudo-color book. No prohibited items were detected." Here, "No prohibited items were detected" is the third label.

[0063] In this embodiment of the application, the second identification data in the pre-description data can be replaced with the target data. In addition, if the third identification data is identified in the description data, a first label can be added to the third identification data. The description data obtained in this way can reflect the false color corresponding to the image (X-ray image) obtained under the illumination of the second type of light source, and can also provide a warning if the image contains contraband, thereby improving the comprehensiveness of the description data.

[0064] In some optional embodiments of this application, step S140, replacing the second identification data in the pre-description data with the target data to obtain description data, includes: replacing the second identification data in the pre-description data with the target data, and adding a second label to the pre-description data to obtain description data, wherein the second label includes type data of the second type of light source.

[0065] For example, the second label can be placed before, after, or in other locations within the descriptive data. The second label may include type data reflecting the type of light source, i.e., the light source data used to capture the image. For instance, the descriptive data could be, "An orange pseudo-color sneaker, a blue pseudo-color camera, a green pseudo-color plastic passport case, and an orange pseudo-color book are visible in the X-ray image." Here, "X-ray image" is the second label.

[0066] It is understandable that the second and third labels can be used together, such as in the description data, "An orange pseudo-color sneaker, a blue pseudo-color camera, a green pseudo-color plastic passport holder, and an orange pseudo-color book are visible in the X-ray image. No contraband was detected." Clearly, the first and second labels can also be used together, such as in the description data, "A blue pseudo-color metal lighter (a contraband) is visible in the X-ray image, surrounded by orange pseudo-color organic matter, including clothing and footwear." Of course, the order in which the first (or third) and second labels are added is not specifically limited in this application; however, for ease of explanation, this embodiment uses the example of adding the first or third label first, followed by the second label, for illustration.

[0067] In some alternative embodiments of this application, the second label is placed before the descriptive data.

[0068] In some alternative embodiments of this application, the first type of light source is natural light. Specifically, when the first type of light source is natural light or artificial light, the second type of light source can be X-rays, infrared light, gamma rays, or other light sources. The artificial light here can include light produced by incandescent lamps, halogen lamps, or LEDs.

[0069] In some alternative embodiments of this application, the image model is one of Qwen2.5-VL, Qwen3-VL, and InternLM-XComposer-2.5.

[0070] In some optional embodiments of this application, replacing the second identification data in the pre-description data with the target data to obtain the description data includes: replacing the second identification data in the pre-description data with the target data and removing the first identification data in the pre-description data to obtain the description data.

[0071] In one example, the first identification data is the material data of each item in the pre-description data. This material data may specifically include the name data of each item, that is, the name data can characterize the material of the item. For example, the pre-description data includes the name data of the item "clothes", which can reflect that the material of the item is an organic material, that is, the name data can be used as material data.

[0072] Optionally, in another example, the first identification data is the material data of each item in the pre-description data. This material data directly represents the specific material of the item. For example, the pre-description data includes "clothing (organic material)," where "(organic material)" is the material data. In this example, the first identification data is used to identify the specific material of each item so as to filter out the false color of each item under X-ray. In the description of the X-ray image, the description of the specific material can be removed, that is, "(organic material)" can be removed.

[0073] like Figure 2 As shown in the figure, this application embodiment provides a model training method, which may include the following:

[0074] Step S210: Obtain a sample training set, which includes sample second-type images and sample description data. The sample second-type images are taken when the target image is photographed under a second-type light source, while the target image is taken under a first-type light source. The first-type light source and the second-type light source are different. The sample description data is description data obtained based on the image description method described above.

[0075] Step S220: Train the model using the sample training set to obtain the second type of image description model.

[0076] The training of the model using the sample training set can be based on qwen3-vl.

[0077] For example, the model could also use CNN (such as ResNet) or ViT for image encoding and Transformer for decoding to generate descriptions.

[0078] In this embodiment, based on the above... Figure 1 The image description method provided in the embodiment obtains sample description data, which can obtain annotation information for the second type of image, i.e., obtain sample description data. This can improve the annotation efficiency and accuracy of samples, thereby improving the training efficiency of the model.

[0079] It should be understood that Figure 2 For details regarding the embodiments, such as the target image, the first type of light source, the second type of light source, and the specific content and corresponding effects of the sample description data, please refer to the above. Figure 1 To avoid repetition, the descriptions in the embodiments will not be repeated here.

[0080] like Figure 3 As shown in the embodiments of this application, another image description method is provided, which may include the following:

[0081] Step S310: Obtain the target second type image.

[0082] A second-type image of a target can be an image taken under a second-type light source. For example, a second-type image of a target can be an X-ray image of the target.

[0083] Step S320: Input the target second-type image into the second-type image description model to generate description data of the target second-type image, wherein the second-type image description model is obtained based on the training method of the above model.

[0084] In this embodiment, by inputting the target second-type image into the second-type image description model, the corresponding description data can be directly obtained, thus improving the efficiency of description data acquisition. Furthermore, since this second-type image description model is... Figure 2 The method provided in the embodiments is used to obtain the data, thus improving the accuracy of the descriptive data.

[0085] It should be understood that Figure 3 For details regarding the embodiments, such as the second type of light source, the second type of image description model, and the specific content and corresponding effects of the description data, please refer to the above. Figure 1 or Figure 2To avoid repetition, the descriptions in the embodiments will not be repeated here.

[0086] This application provides an image description device, such as... Figure 4 As shown, the first image description device 400 includes: a first acquisition module 410, used to acquire a target image, which is captured under a first type of light source; an extraction module 420, used to extract features from the target image using an image model to obtain pre-description data, which includes a set of textual description information for each item in the target image; an identification module 430, used to identify first identification data in the pre-description data, and obtain target data corresponding to the first identification data in the transformation set through a transformation set, wherein the first identification data is the material data of each item in the pre-description data, and the transformation set includes a mapping set between the material data and the image pseudo-color data of the material corresponding to the material data obtained under the illumination of the second type of light source, wherein the target data is pseudo-color data, and the first type of light source and the second type of light source are different; and a replacement module 440, used to replace the second identification data in the pre-description data with the target data to obtain description data, wherein the second identification data is the color data of each item in the pre-description data.

[0087] In this embodiment, descriptions of the same photographed object under the same environment can be generated based on images with relevant image model light source types. For example, if an X-ray image description is required, natural light photography can be added before or after the X-ray imaging point, ensuring the natural light image and X-ray image have the same photographed object and environment. Image model analysis is performed on the obtained natural light image to obtain a natural light description. Based on the different pseudocolors of X-ray images of different materials, the material of each item in the natural light description is analyzed, and different materials and their corresponding pseudocolors are filtered out. Then, these pseudocolors are used to replace the colors of each item in the natural light description, thus obtaining the X-ray image description. In the aforementioned process, the image description originates from a natural light image model. This type of model is relatively mature, with many models available, resulting in high description accuracy and facilitating multi-faceted verification. It enables rapid description of X-ray images, improving the low efficiency and inconsistency of manual description. Furthermore, the description obtained from the natural light image model used in the description process is still the description corresponding to the natural light image, reducing the difficulty of describing direct cross-modal differences. Thus, the generation efficiency and reliability of description data for images obtained under second-type light source illumination can be improved.

[0088] According to one embodiment of this application, the second type of light source includes X-rays.

[0089] According to one embodiment of this application, the replacement module 440 is used to: when the existence of third identification data in the description data is detected, add a first label to the third identification data to obtain the description data, wherein the third identification data includes security inspection prohibited item name data, and the first label includes prohibited item entries marked at the third identification data.

[0090] According to one embodiment of this application, the replacement module 440 is used to: replace the second identification data in the pre-description data with the target data, and add a second label to the pre-description data to obtain description data, wherein the second label includes type data of the second type of light source.

[0091] According to one embodiment of this application, the first type of light source is natural light.

[0092] According to one embodiment of this application, the image model is one of Qwen2.5-VL, Qwen3-VL, and InternLM-XComposer-2.5.

[0093] It should be understood that the operation and function of the first acquisition module 410, extraction module 420, identification module 430, and replacement module 440 in the above embodiments can be referred to the above. Figure 1 The descriptions of the methods provided in the embodiments will not be repeated here to avoid repetition.

[0094] This application provides a model training device, such as... Figure 5 As shown, the training device 500 for the model includes: a second acquisition module 510, used to acquire a sample training set, which includes sample second-type images and sample description data, wherein the sample second-type images are taken by the object of the target image under a second-type light source, and the target image is taken under a first-type light source, the first-type light source and the second-type light source are different, and the sample description data is description data obtained based on the above-mentioned image description method; and a training module 520, used to train the model using the sample training set to obtain a second-type image description model.

[0095] In this embodiment, based on the above... Figure 2 The image description device provided in the embodiment obtains sample description data and can obtain annotation information for the second type of image, that is, obtain sample description data. This can improve the annotation efficiency and accuracy of the samples, thereby improving the training efficiency of the model.

[0096] It should be understood that the operation and function of the second acquisition module 510 and the training module 520 in the above embodiments can be referred to the above. Figure 2 The descriptions of the methods provided in the embodiments will not be repeated here to avoid repetition.

[0097] This application provides another image description device, such as... Figure 6 As shown, the second image description device 600 includes: a third acquisition module 610 for acquiring a target second type image; and an input module 620 for inputting the target second type image into a second type image description model to generate description data of the target second type image, wherein the second type image description model is obtained based on the training method of the above-mentioned model.

[0098] In this embodiment, by inputting the target second-type image into the second-type image description model, the corresponding description data can be directly obtained, thus improving the efficiency of description data acquisition. Furthermore, since this second-type image description model is... Figure 2 The method provided in the embodiments is used to obtain the data, thus improving the accuracy of the descriptive data.

[0099] It should be understood that the operation and function of the third acquisition module 610 and the input module 620 in the above embodiments can be referred to the above. Figure 3 The descriptions of the methods provided in the embodiments will not be repeated here to avoid repetition.

[0100] This application provides an electronic device, such as... Figure 7 As shown, the electronic device 700 includes: a processor 701, a memory 702, and a communication interface 703; the memory 702 and the communication interface 703 are coupled to the processor 701. The memory 702 is used to store computer program code, which includes computer instructions. When the processor 701 executes the computer instructions, the electronic device performs the aforementioned image description method and model training method. It should be understood that this application does not limit the number of processors 701 and memory 702 in the electronic device.

[0101] In some optional embodiments of this application, the electronic device further includes a bus 704, through which the processor 701, the memory 702, and the communication interface 703 communicate.

[0102] The 704 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The 704 bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus 704 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 704 may include a path for transmitting information between various components of an electronic device (e.g., memory 702, processor 701, communication interface 703).

[0103] The processor may be any one or more of the following: a central processing unit, a graphics processing unit, a microprocessor (MP), or a digital signal processor (DSP).

[0104] Memory 702 may include volatile memory, such as random access memory (RAM). Memory 702 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0105] The communication interface 703 uses a command distribution module, such as, but not limited to, a network interface card or transceiver, to enable communication between electronic devices and other devices or communication networks.

[0106] In some embodiments of this application, a computer-readable storage medium is provided, which stores a computer program. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the image description method and the model training method described above.

[0107] In some embodiments of this application, an apparatus (e.g., a chip system) is provided, which includes a processor for supporting electronic devices in implementing the above-described image description method and model training method.

[0108] In some alternative embodiments of this application, the device further includes a memory for storing program instructions and data necessary for the electronic device. When the device is a chip system, it may be composed of chips or may include chips and other discrete components.

[0109] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0110] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0115] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program verification codes, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0116] It should be noted that in the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0118] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image description method, characterized in that, include: Acquire a target image, which is captured under a first type of light source; Feature extraction is performed on the target image using an image model to obtain pre-description data, which includes a set of textual description information for each item in the target image. The first identification data in the pre-description data is identified, and the target data corresponding to the first identification data in the conversion set is obtained through the conversion set based on the first identification data. The first identification data is the material data of each item in the pre-description data. The conversion set includes a mapping set between the material data and the image pseudo-color data of the material corresponding to the material data obtained under the illumination of the second type of light source. The target data is pseudo-color data. The first type of light source and the second type of light source are different. The second identification data in the pre-description data is replaced with the target data to obtain description data. The second identification data is the color data of each item in the pre-description data. The description data is used to train the model with the corresponding second type image to obtain the second type image description model. The second type image is the photograph of the target image corresponding to the description data under the second type light source.

2. The image description method according to claim 1, characterized in that, The second type of light source includes X-rays.

3. The image description method according to claim 1, characterized in that, The step of replacing the second identification data in the pre-description data with the target data to obtain the description data includes: If a third identification data is detected in the description data, a first label is added to the third identification data to obtain the description data. The third identification data includes the name data of prohibited items for security checks, and the first label includes the prohibited item entry marked on the third identification data.

4. The image description method according to claim 1, characterized in that, The step of replacing the second identification data in the pre-description data with the target data to obtain the description data includes: The second identification data in the pre-description data is replaced with the target data, and a second label is added to the pre-description data to obtain the description data, wherein the second label includes the type data of the second type of light source.

5. The image description method according to claim 1, characterized in that, The first type of light source is natural light.

6. The image description method according to any one of claims 1-5, characterized in that, The image model is one of Qwen2.5-VL, Qwen3-VL, and InternLM-XComposer-2.

5.

7. A method for training a model, characterized in that, include: A sample training set is obtained, the sample training set including sample second type images and sample description data, wherein the sample second type images are taken by the object of the target image under a second type of light source, the target image is taken under a first type of light source, the first type of light source and the second type of light source are different, and the sample description data is description data obtained based on the image description method of any one of claims 1-6; The model is trained using the sample training set to obtain the second type of image description model.

8. An image description method, characterized in that, include: Obtain the target second type image; The target second-type image is input into the second-type image description model to generate description data of the target second-type image, wherein the second-type image description model is obtained based on the training method of the model described in claim 7.

9. An image description device, characterized in that, include: An acquisition module is used to acquire a target image, which is captured under a first type of light source; The extraction module is used to extract features from the target image using an image model to obtain pre-description data, wherein the pre-description data includes a set of textual description information for each item in the target image; The identification module is used to identify the first identification data in the pre-description data, and obtain the target data corresponding to the first identification data in the transformation set through the transformation set based on the first identification data. The first identification data is the material data of each item in the pre-description data. The transformation set includes a mapping set between the material data and the image pseudo-color data of the material corresponding to the material data obtained under the illumination of the second type of light source. The target data is pseudo-color data. The first type of light source and the second type of light source are different. The replacement module is used to replace the second identification data in the pre-description data with the target data to obtain description data. The second identification data is the color data of each item in the pre-description data. The description data is used to train the model with its corresponding second type image to obtain a second type image description model. The second type image is the photograph of the target image corresponding to the description data under a second type of light source.

10. A training device for a model, characterized in that, include: An acquisition module is used to acquire a sample training set, the sample training set including sample second type images and sample description data, wherein the sample second type images are taken by the object of the target image under the second type of light source, the target image is taken under the first type of light source, the first type of light source and the second type of light source are different, and the sample description data is description data obtained based on the image description method of any one of claims 1-6; The training module is used to train the model using the sample training set to obtain a second type of image description model.

11. An electronic device, characterized in that, The electronic device includes: Processor, memory, and communication interface; The memory and the communication interface are coupled to the processor. The memory is used to store computer program code, which includes computer instructions. Wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1-6, or the method as described in claim 7 or claim 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, perform the method as described in any one of claims 1-6, or the method as described in claim 7 or claim 8.

Images