Infrared-optical fusion scene semantic description method and device based on large model

Abstract

The application provides an infrared-optical fusion scene semantic description method and device based on a large model, and belongs to the technical field of image description and computer vision. The method comprises the following steps: inputting a visible light image pair and an infrared image pair obtained to a preset scene description model to obtain an initial scene description sentence of the image pair; inputting the infrared image of the image pair to a preset target detection model to obtain the category of each target detection frame corresponding target in the infrared image, and the center point coordinates and relative size of the target; inputting the above information to a large language model with prompt optimization completed, and the large language model outputs a final scene description sentence corresponding to the image pair. The target detection technology and the scene description technology are combined through the large language model technology, the defects that the previous image description model cannot be applied to a dark light scene or is not accurate in description can be overcome, and especially, the scene description can be generated accurately and in detail under insufficient light, so that the application value is high.

Description

Technical Field

[0001] This invention belongs to the fields of image description and computer vision technology, and specifically relates to a method and apparatus for semantic description of infrared-optical fusion scenes based on a large model. Background Technology

[0002] Vision is one of the most important ways for humans to acquire information about the world around them. With the continuous advancement of machine learning and computer vision technologies, scene description technology has gradually become an important means of obtaining visual information from the outside world. Scene description technology is an interdisciplinary technology that combines computer vision and natural language processing. Its goal is to enable computers to understand and describe the content of images, thereby achieving seamless interaction between images and natural language. Scene description technology can describe the content of scenes in images in detail, helping users better understand their surroundings and providing safeguards for human health and well-being, such as providing security monitoring in public places and guidance and assistance to visually impaired individuals. The development and application of scene description technology have allowed more people to experience the convenience brought by information technology.

[0003] Current main image description methods include: 1) CNN-RNN (Convolutional Neural Network-Recurrent Neural Network) structure model: This is the most classic image description model. It inputs the image into a convolutional neural network for feature extraction, and then inputs the extracted features into a recurrent neural network (usually LSTM, Long Short-Term Memory network) to generate a natural language description of the image content. 2) Attention mechanism model: To improve the traditional CNN-RNN structure model, an attention mechanism has been introduced. Through the attention mechanism, the model can better focus on different regions in the image and generate more accurate descriptions. This method allows the model to dynamically adjust the description generation process according to the importance of different regions in the image. 3) Transformer model: With the successful application of the Transformer model in natural language processing tasks, it has also been introduced into image description tasks. Through the self-attention mechanism, the Transformer model can globally model different parts of the image and generate more accurate and fluent descriptions. 4) Fine-tuning of pre-trained models: With the help of pre-trained models on large-scale image-text datasets (such as OpenAI's CLIP model), classification tasks can be performed through fine-tuning, and more complex image description tasks can be handled, such as generating text related to images.

[0004] However, these techniques are typically geared towards visible light environments; in nighttime or low-light conditions, the available visible light information is severely lacking, resulting in poor scene description. To address this, some have proposed extracting information from infrared images to generate descriptions, but infrared images lack detail, leading to poor accuracy in scene description. Furthermore, some have proposed fusing features from infrared and visible light images for image description generation, but this process inevitably results in the loss of unique infrared and visible light features, leaving the description quality still unsatisfactory. Therefore, current scene description methods cannot adequately meet the needs of users in special environments. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an infrared-optical fusion scene semantic description method and device based on a large model. This invention combines target detection technology and scene description technology through LLM (Large Language Model) technology, overcoming the limitations of previous image description models that are unsuitable for low-light scenes or provide inaccurate descriptions. Especially under insufficient light conditions, it can generate accurate and detailed scene descriptions, demonstrating high application value.

[0006] A first aspect of this invention proposes a method for semantic description of infrared-optical fusion scenes based on a large model, comprising:

[0007] A visible light image and an infrared image from the same scene are acquired and matched. The matched visible light image and infrared image are then combined to form a visible light and infrared image pair to be processed.

[0008] The visible light and infrared image pairs to be processed are input into a preset scene description model, and the scene description model outputs an initial scene description statement for the image pairs;

[0009] The infrared image from the visible light and infrared image pair to be processed is input into a preset target detection model. The target detection model outputs the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box, thereby obtaining the center point coordinates and relative size of the target corresponding to each target detection box.

[0010] The large language model is optimized by taking the initial scene description statement of the image pair, the type of target corresponding to each target detection box, the center point coordinates and relative size of the target corresponding to each target detection box, and the input prompts. The large language model outputs the final scene description statement corresponding to the visible light and infrared image pair to be processed.

[0011] In one specific embodiment of the present invention, the scene description model adopts an RGB-IR multimodal fusion image description model.

[0012] In one specific embodiment of the present invention, the target detection model adopts the YOLOv7 target detection model.

[0013] In one specific embodiment of the present invention, before inputting the visible light and infrared images to be processed into a preset scene description model, the method further includes:

[0014] Train the scene description model;

[0015] Training the scene description model includes:

[0016] 1) Using a paired visible light camera and an infrared camera, visible light images and corresponding infrared images of the same type of scene are acquired to obtain multiple sets of visible light and infrared image pairs, wherein the category of the scene is consistent with the category of the scene corresponding to the visible light and infrared image pair to be processed;

[0017] 2) For each pair of visible light and infrared images obtained in step 1), scene description statements are annotated. The annotation content of the two images in each pair of visible light and infrared images is the same. The scene description statement reflects the target and its state in the scene in which the image is located.

[0018] After the annotation is completed, each pair of visible light and infrared images and its corresponding scene description statement constitutes an image description training sample, and all image description training samples constitute the image description training set.

[0019] 3) Use the image description training set obtained in step 2) to train the scene description model, and obtain the trained scene description model.

[0020] In one specific embodiment of the present invention, before inputting the infrared image from the visible light and infrared image pair to be processed into a preset target detection model, the method further includes:

[0021] Train the target detection model;

[0022] Training the target detection model includes:

[0023] 1) Obtain an infrared image training set consisting of multiple labeled infrared images, wherein the scene category corresponding to each infrared image in the infrared image training set is consistent with the scene category corresponding to the image description training set, and each infrared image in the infrared image training set is labeled with the type of target corresponding to each target detection box in the image and the position information of the target detection box;

[0024] 2) Train the target detection model using the infrared image training set obtained in step 1) to obtain the trained target detection model.

[0025] In one specific embodiment of the present invention, the method further includes:

[0026] 1) Input each pair of visible light and infrared images from the image description training set into the trained scene description model, and the scene description model outputs a scene description statement for the image pair;

[0027] 2) Input each infrared image from the infrared image training set into the trained target detection model. The target detection model outputs the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box. The position information of the target detection box includes the coordinates of the upper left corner vertex and the lower right corner vertex of each target detection box. Then, calculate the center point coordinates and relative size of the target corresponding to each target detection box in each infrared image of the infrared image training set, as shown in the following expression:

[0028]

[0029] In the formula, x1 is the x-coordinate of the top left corner vertex of the target detection box, y1 is the y-coordinate of the top left corner vertex of the target detection box, x2 is the x-coordinate of the bottom right corner vertex of the target detection box, y2 is the y-coordinate of the bottom right corner vertex of the target detection box, x0 is the x-coordinate of the center point of the target detection box, y0 is the y-coordinate of the center point of the target detection box, and s is the relative size of the target.

[0030] 3) Using the image description training set, the infrared image training set, and the results of steps 1) and 2), suggestions are made to optimize the large language model so that the large language model outputs a scene description statement that integrates multiple information corresponding to the input image scene.

[0031] In one specific embodiment of the present invention, the suggestion to optimize the large language model includes:

[0032] By combining the information output by the scene description model and the object detection model, a task description is designed for the large language model;

[0033] Based on prior knowledge, rules are set for the large language model to meet user needs;

[0034] Provide example text that conforms to the rules for the large language model.

[0035] A second aspect of this invention provides a scene semantic description device based on a large model of infrared-optical fusion, comprising:

[0036] The image acquisition module is used to acquire a visible light image and an infrared image of the same scene respectively and match them, and combine the matched visible light image and infrared image into a visible light and infrared image pair to be processed.

[0037] An initial scene description module is used to input the visible light and infrared image pairs to be processed into a preset scene description model, and the scene description model outputs an initial scene description statement for the image pairs.

[0038] The target detection module is used to input the infrared image from the visible light and infrared image pair to be processed into a preset target detection model. The target detection model outputs the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box, thereby obtaining the center point coordinates and relative size of the target corresponding to each target detection box.

[0039] The large language model description module is used to optimize the large language model by taking the initial scene description statement of the image pair, the type of the target corresponding to each target detection box, the center point coordinates and relative size of the target corresponding to each target detection box, and the large language model outputs the final scene description statement corresponding to the visible light and infrared image pair to be processed.

[0040] A third aspect of the present invention provides an electronic device comprising:

[0041] At least one processor; and a memory communicatively connected to said at least one processor;

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-described infrared-optical fusion scene semantic description method based on a large model.

[0043] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the above-described infrared-optical fusion scene semantic description method based on a large model.

[0044] The features and beneficial effects of this invention are as follows:

[0045] This invention combines target detection technology and scene description technology using LLM (Large Language Model) technology. When the scene description model performs poorly in low-light conditions, an infrared-based target detection model detects the type, quantity, and relative position of targets. Then, leveraging the powerful semantic understanding and reasoning capabilities of the large language model, the information obtained from both methods is semantically fused to generate a more accurate and detailed description. This invention fully utilizes the advantages of each technology, improving the accuracy of scene understanding and description. Specifically, the infrared-based target detection model can detect the type and quantity of targets in low-light environments, thus compensating for the impact of insufficient light on model accuracy and improving description accuracy. By combining target detection results with the semantic understanding and reasoning capabilities of the large language model, this invention can generate more accurate and detailed scene descriptions.

[0046] This invention can automatically analyze and describe scene features in infrared images, providing a wide range of application possibilities in various fields. For example, it can be applied to the military field, helping the army conduct nighttime reconnaissance, target identification, and surveillance missions, improving combat efficiency and security; in the security field, it can be used in intelligent monitoring systems to promptly detect anomalies and ensure public safety; and in the public service field, it can help blind people identify surrounding objects and obstacles, improving their quality of life and independence. Attached Figure Description

[0047] Figure 1 This is an overall flowchart of the infrared-optical fusion scene semantic description method based on a large model according to an embodiment of the present invention. Detailed Implementation

[0048] This invention proposes a method and apparatus for semantic description of infrared-optical fusion scenes based on a large model. The invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] This embodiment proposes a scene semantic description method based on a large model of infrared-optical fusion. The overall process is as follows: Figure 1 As shown, it includes:

[0050] A visible light image and an infrared image from the same scene are acquired and matched. The matched visible light image and infrared image are then combined to form a visible light and infrared image pair to be processed.

[0051] The visible light and infrared image pairs to be processed are input into a preset scene description model, and the scene description model outputs an initial scene description statement for the image pairs;

[0052] The infrared image from the visible light and infrared image pair to be processed is input into a preset target detection model. The target detection model outputs the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box, thereby obtaining the center point coordinates and relative size of the target corresponding to each target detection box.

[0053] The large language model is optimized by taking the initial scene description statement of the image pair, the type of target corresponding to each target detection box, the center point coordinates and relative size of the target corresponding to each target detection box, and the input prompts. The large language model outputs the final scene description statement corresponding to the visible light and infrared image pair to be processed.

[0054] In a specific embodiment of the present invention, the infrared-optical fusion scene semantic description method based on a large model includes: a data preparation stage, a design cue word stage, and a usage stage, comprising the following steps:

[0055] (1) Data preparation stage; the specific steps are as follows:

[0056] (1-1) Acquire multiple image pairs consisting of visible light images and infrared images.

[0057] In this embodiment, a paired visible light camera and an infrared camera are used to acquire visible light images and corresponding infrared images of the same type of scene, resulting in multiple pairs of visible light and infrared images. This invention has no special requirements for image resolution. In one specific embodiment, the visible light camera and the infrared camera each acquired 2600 visible light images and 2600 infrared images, forming 2600 pairs of visible light and infrared images.

[0058] In one specific embodiment of the present invention, the scene selected is a conventional traffic scene such as a campus or urban road, and the image resolution is 640×512 pixels.

[0059] (1-2) Annotate each pair of visible light and infrared images obtained in step (1-1) with scene description statements. The annotations of the two images in each pair of visible light and infrared images are consistent. The scene description statement reflects the target and its state in the scene in which the image is located.

[0060] In one specific embodiment of the present invention, the annotation objects of the scene description statements mainly include three categories of targets: people, bicycles, and cars. The annotation statement is as follows: "A person is riding a bicycle on the side of the road." In this embodiment, each image pair obtains 5 manually annotated scene description statements. Finally, each pair of visible light and infrared images and its corresponding scene description statements constitutes an image description training sample, and all image description training samples constitute an image description training set.

[0061] (1-3) Use the image description training set obtained in step (1-2) to train the scene description model and obtain the trained scene description model.

[0062] In this embodiment, the scene description model adopts the existing RGB-IR multimodal fusion image description model.

[0063] In one specific embodiment of the present invention, the model training parameters are: batch_size = 32 (16 image pairs are trained per batch), epoch = 200 (the network is trained for 200 epochs), the initial learning_rate is set to 5e-4, and the learning_rate_decay_start is set to 50 (the learning rate decays starting from the 50th epoch).

[0064] (1-4) Input each pair of visible light and infrared images from the image description training set into the scene description model trained in step (1-3). The scene description model outputs a scene description statement for the image pair, such as: There is a car driving on the road.

[0065] (1-5) Obtain the infrared image training set.

[0066] In this embodiment, an infrared image training set is obtained, consisting of multiple labeled infrared images. Each infrared image in the training set corresponds to a scene of the same type as the scene in the image description training set. The size of each infrared image in the training set is not specifically required and can differ from the image size in the image description training set. The annotation content for each infrared image in the training set includes the type of target corresponding to each target detection box in the image and the location information of the target detection box.

[0067] In one specific embodiment of the present invention, a publicly labeled FLIR infrared dataset is selected. The infrared images in this dataset are acquired by infrared cameras installed on vehicles. The dataset mainly collects target information in traffic scenes. The image size is 640×512 pixels. Each image contains a target detection box. The target categories mainly include pedestrians, bicycles, cars, etc.

[0068] (1-6) Use the infrared image training set obtained in step (1-5) to train the target detection model and obtain the trained target detection model.

[0069] In one specific embodiment of the present invention, the target detection model adopts the existing YOLOv7 target detection model.

[0070] In this embodiment, the model training parameters are: batch_size = 16 (16 images per batch), epoch = 300 (300 training epochs), optimizer type is SGD, momentum = 0.937 (optimizer internal parameter), weight_decay = 5e-4 (weight decay), and the maximum learning rate is set to 0.01. Finally, the trained YOLOv7 object detection model is obtained. The object detection model takes an infrared image as input and outputs the corresponding object detection box information in the image.

[0071] (1-7) Input each infrared image in the infrared image training set into the target detection model trained in step (1-6). The target detection model can output the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box. The position information of the target detection box includes the coordinates of the upper left corner vertex and the lower right corner vertex of each target detection box.

[0072] (1-8) Process the target detection box position information output in step (1-7). Specifically, using the target detection box position information, calculate the center point coordinates and relative size of the target corresponding to the detection box, as shown in the following expression:

[0073]

[0074] In the formula, x1 is the x-coordinate of the top left corner vertex of the target detection box, y1 is the y-coordinate of the top left corner vertex of the target detection box, x2 is the x-coordinate of the bottom right corner vertex of the target detection box, y2 is the y-coordinate of the bottom right corner vertex of the target detection box, x0 is the x-coordinate of the center point of the target detection box, y0 is the y-coordinate of the center point of the target detection box, and s is the relative size of the target.

[0075] After processing, the target type, center point coordinates, and relative size information of each target detection box corresponding to each infrared image in the infrared image training set are finally obtained. For example, for a car: Car 1: Center point coordinates = (200, 300), relative size = 45.

[0076] (2) Design prompt words stage.

[0077] In this embodiment, LLM (Large Language Models) refers to a class of language models with a huge number of parameters, typically hundreds of millions to trillions. This enables them to capture and learn complex language patterns and relationships, thus performing exceptionally well on various natural language processing tasks. LLMs possess powerful generation and understanding capabilities, capable of generating coherent and logical text sequences based on the input context, and also able to understand and process various semantic and syntactic structures of human language.

[0078] A cue word is a design pattern used to guide an LLM (Limited Language Management) to perform a specific task or generate a specific type of output. The relationship between an LLM and cue words is that cue words provide a way to interact with an LLM, guiding it to perform various tasks and generate specific types of text output. The design quality and selection of cue words are crucial to the performance and output of an LLM; a good cue word can significantly improve the performance of an LLM on a specific task.

[0079] The core principle of cue-guided Large Language Models (LLMs) is to provide an input method that guides the model to generate a specific type of output. This input method is typically presented in natural language, giving a description, requirement, or condition for the desired output. Cue words can be one or more sentences that explicitly instruct the model on the task it should perform or the requirements for the generated text.

[0080] The key to implementing prompt-guided LLM lies in designing appropriate prompts and ensuring that the model can correctly understand and meet the requirements of those prompts. This requires a deep understanding of the characteristics and requirements of the task. Simultaneously, it necessitates debugging and optimization for specific application scenarios to achieve the best performance and user experience.

[0081] In this embodiment, the design prompts mainly consist of three parts: a design task description, design task rules, and design guidance example text; the specific steps are as follows:

[0082] (2-1) Design task description.

[0083] In this embodiment, the task explained to the LLM is to integrate known information (i.e., scene information obtained by the image description model and the object detection model) to help the user analyze the scene and generate more accurate scene description statements.

[0084] In one specific embodiment of this invention, the specific prompts in the task description are: "You are an excellent visual AI assistant, skilled at analyzing scenes in images. The image size is 640×512 pixels. I will provide you with an initial rough description of the scene, as well as the types, quantities, center point coordinates, and relative sizes of all targets in the scene. Initial rough description of the scene: (results generated by the image description model); Target information detected in the scene: (results generated by target detection); Your task is to synthesize the above information to help the user analyze the scene in the image and generate a more accurate scene description."

[0085] (2-2) Design task rules.

[0086] In this embodiment, prior knowledge based on common sense is used to set rule texts for LLM that meet user needs, and more reasonable text content is generated based on these rules.

[0087] In one specific embodiment of the present invention, rule 1 states that in actual road scenarios, there are pedestrians, bicycles placed on the roadside, and cyclists; however, the selected infrared image training set classifies people and bicycles, and the output results after image detection will also transmit the two types of information separately to the LLM. This results in the fact that in reality, people are riding bicycles, but the final result describing people and bicycles separately is obviously inappropriate. By summarizing and generalizing these problems, we found that there are patterns in the target detection bounding boxes when a person is riding a bicycle. Combining common sense and the logical reasoning ability of LLM, we defined the rules as follows: "If the detected targets in the scene of the image include people and bicycles, then the x-coordinate of the center point of each bicycle must be compared with the x-coordinate of the center point of the person. If the difference in the x-coordinates of the center points is no greater than 5, it can be inferred that there is a relationship between the person and the bicycle, that is, the person is riding a bicycle. If the difference in the x-coordinates exceeds 5, it can be determined that the person and the bicycle are not related, that is, the person is not riding a bicycle. For example: if the detected targets in the scene include 3 people and 2 bicycles, then the x-coordinates of the center points of the two bicycles need to be compared with the other three people. If the difference in the x-coordinates of the center points of a person and a bicycle is 3, which is no more than 5, it can be inferred that the person is riding a bicycle. If the difference in the x-coordinates of the center pixels of another person and another bicycle is 25, which is more than 5, then it can be inferred that there is no relationship between them, that is, the person is not riding a bicycle."

[0088] Rule 2 Explanation: Using the relative size *s* of scene targets obtained after processing by the object detection model, targets are categorized into three sizes: large, medium, and small. Based on the principle that the size decreases with distance in an image, the distance of the target is further inferred. Specific prompts are: "If the relative size of some detected targets in the scene is less than 32, it can be inferred that these targets are located far away. If the relative size of a target is greater than 96, it is determined that it is located very close. If the relative size of a target is between 32 and 96, it is located in a moderately close position. For example, if there are three people in the scene, one person's relative size is 22.3, less than 32, so it can be inferred that they are far from the user; another person's relative size is 74.5, between 32 and 96, so it can be determined that they are at a moderately close distance to the user; and the third person's relative size is 100, greater than 96, so they are close to the user."

[0089] Rule 3 Explanation: The relative position of the target in the scene is inferred based on the target center point coordinates (x0, y0) obtained by the target detection model. The specific prompt is as follows: "Since the scene image size is 640×512 pixels, the range of x-coordinates of the target center point is defined as follows: 280-360 as the middle region, 150-280 as the left region, 360-490 as the right region, 0-150 as the left edge region, and 490-640 as the right edge region. Based on the given target center point coordinates, the region where the target is located and its position relative to the user can be inferred. For example, (404.5, 204.0) is in the right region of the image, so the target is in front of the user's right; (229.5, 193.0) is in the left region of the image, so the target is in front of the user's left; (288.0, 190.5) is in the middle region of the image, so the target is in front of the user; (105.0, 241.0) is in the left edge region of the image, so the target is in front of the user's left; (580.5, 255.0) is in the right edge region of the image, so the target is in front of the user's right."

[0090] (2--3) Design guiding sample text.

[0091] In this embodiment, specific examples or guiding text are provided for LLM to more clearly illustrate the task requirements and expected outputs. These examples can be template text used as a reference for model generation.

[0092] In a specific embodiment of the present invention, the specific prompt word is: "Based on the above rules, here is an example of analyzing scene semantic information: Initial rough description of the scene: A person is walking on the road. Target information detected in the scene: 3 people, 3 bicycles, 1 car. Person 1: Center point coordinates = (404.5, 204.0), relative size = 85.5; Person 2: Center point coordinates = (451.0, 198.0), relative size = 110.6; Person 3 ... Bicycle 1: Center point coordinates = (229.0, 193.0), relative dimension = 14.5; Bicycle 2: Center point coordinates = (230.0, 190.5), relative dimension = 15.2; Bicycle 3: Center point coordinates = (405.0, 241.0), relative dimension = 67.3; Bicycle 4: Center point coordinates = (105.0, 241.0), relative dimension = 35.1; Car 1: Center point coordinates = (605.0, 241.0), relative dimension = 20.6.

[0093] According to rule 1:

[0094] Since the target includes people and bicycles, we compare the x-coordinates of the center points of the three bicycles with the x-coordinates of the center points of the three people. We can observe that the difference in x-coordinate between bicycle 1 and person 3 is 1, which is less than 5, so we can conclude that person 3 is riding bicycle 1. Similarly, the difference in x-coordinate between bicycle 2 and person 1 is 0.5, which is less than 5, so we can also conclude that person 1 is riding bicycle 2. The difference in x-coordinate between bicycle 3 and person 2 exceeds 5, so we can infer that bicycle 3 and person 2 are unrelated. In conclusion: person 1 is riding bicycle 2, person 3 is riding bicycle 1, bicycle 3 is parked in one place, and person 2 is walking in another place.

[0095] According to rule 2:

[0096] The relative dimensions of person 1 and bicycle 2 are both between 32.0 and 96.0, indicating that the distance between person 1 riding bicycle 2 and the user is moderate. The relative dimension of person 2 is greater than 96.0, indicating that person 2 is close to the user. The relative dimensions of person 3 and bicycle 1 are both less than 32.0, indicating that person 3 riding bicycle 1 is relatively far from the user. The relative dimension of bicycle 3 is between 32.0 and 96.0, therefore its parking location is moderately far from the user. The relative dimension of bicycle 1 is less than 32.0, indicating that bicycle 1 is relatively far from the user.

[0097] According to rule 3:

[0098] The x-coordinates of Person 1 and Bicycle 2 are 404.5 and 405.0 respectively, both within the range of 360-490, belonging to the right-side region, indicating that Person 1, riding Bicycle 2, is located to the right and slightly in front of the user. Person 2's x-coordinate is 451.0, also within the range of 360-490, belonging to the right-side region, indicating that Person 2 is also to the right and slightly in front of the user. The x-coordinates of Person 3 and Bicycle 1 are 229.0 and 230.0 respectively, both within the range of 150-280, belonging to the left-side region, indicating that Person 3, riding Bicycle 1, is located to the left and slightly in front of the user. The x-coordinate of Bicycle 3 is 105.0, ranging from 0-150, belonging to the left edge region, indicating that Bicycle 3 is located to the left and slightly in front of the user. The x-coordinate of Car 1 is 605.5, within the range of 490-640, belonging to the right edge region, indicating that Car 1 is to the right and slightly in front of the user.

[0099] In summary, based on the above rules: Person 1 riding bicycle 2 is to the user's right front, at a moderate distance. Person 2 is to the user's right front, very close. Person 3 riding bicycle 1 is to the user's left front, relatively far away. Bicycle 3 is parked to the user's left front, at the left edge of the scene, at a moderate distance. Car 1 is to the user's right front, at the right edge of the scene, relatively far away.

[0100] Describe the current scene to the user in concise text: "Hello, there is someone very close to you on your right front; there is someone riding a bicycle at a moderate distance to you on your right front; there is another bicycle at a moderate distance to you, located at the left edge of the scene. Additionally, there is a car at the right edge of the scene, farther away, and someone riding a bicycle on your left front, also farther away."

[0101] Furthermore, in this embodiment, after designing the prompt words, testing is conducted. The effectiveness of the prompt words is evaluated using sample data and manual verification. Based on the test results, the design of the prompt words is repeatedly adjusted and optimized until the expected effect is achieved. The specific prompt words mentioned above are all the best results obtained after multiple adjustments and optimizations.

[0102] (3) Usage stage.

[0103] In this embodiment, the usage phase utilizes the scene description of the provided image output by the large language model that has been optimized during the design prompt word phase; the specific steps are as follows:

[0104] (3-1) Obtain a visible light image and an infrared image in the same scene and match them. Combine the matched visible light image and infrared image to form a visible light and infrared image pair.

[0105] In this embodiment, the visible light and infrared image pairs are acquired by a matched visible light camera and an infrared camera. The image pairs belong to the same scene category and have the same image resolution as the image pairs in the image description training set.

[0106] (3-2) Input the image pair obtained in step (3-1) into the trained scene description model, and the scene description model outputs a scene description statement for the image pair.

[0107] (3-3) Input the infrared image from the image pair obtained in step (3-1) into the trained target detection model. The target detection model outputs the type of target corresponding to each target detection box in the infrared image and the location information of the target detection box.

[0108] Then, using the target detection box position information, the center point coordinates and relative size of the target corresponding to each target detection box are calculated.

[0109] (3-4) Input the results of step (3-2) and step (3-3) into the large language model that has been optimized as indicated in step (2). The large language model outputs the image scene description statement after multimodal semantic fusion.

[0110] In this invention, all calls to the LLM are performed in a Python 3.8 environment, and the API calls to GPT-3.5 are made through the function library provided by OpenAI to realize the information transmission between the user and the LLM.

[0111] To implement the above embodiments, a second aspect of the present invention proposes an infrared-optical fusion scene semantic description device based on a large model, comprising:

[0112] The image acquisition module is used to acquire a visible light image and an infrared image of the same scene respectively and match them, and combine the matched visible light image and infrared image into a visible light and infrared image pair to be processed.

[0113] An initial scene description module is used to input the visible light and infrared image pairs to be processed into a preset scene description model, and the scene description model outputs an initial scene description statement for the image pairs.

[0114] The target detection module is used to input the infrared image from the visible light and infrared image pair to be processed into a preset target detection model. The target detection model outputs the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box, thereby obtaining the center point coordinates and relative size of the target corresponding to each target detection box.

[0115] The large language model description module is used to optimize the large language model by taking the initial scene description statement of the image pair, the type of the target corresponding to each target detection box, the center point coordinates and relative size of the target corresponding to each target detection box, and the large language model outputs the final scene description statement corresponding to the visible light and infrared image pair to be processed.

[0116] It should be noted that the foregoing explanation of the embodiment of the infrared-optical fusion scene semantic description method based on a large model also applies to the infrared-optical fusion scene semantic description device based on a large model in this embodiment, and will not be repeated here. According to an embodiment of the present invention, an infrared-optical fusion scene semantic description device based on a large model acquires a visible light image and an infrared image of the same scene and matches them to form a pair of visible light and infrared images to be processed. The pair of visible light and infrared images to be processed is input into a preset scene description model, which outputs an initial scene description statement for the image pair. The infrared image in the pair of visible light and infrared images to be processed is input into a preset target detection model, which outputs the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box, thereby obtaining the center point coordinates and relative size of the target corresponding to each target detection box. The initial scene description statement of the image pair, the type of target corresponding to each target detection box, the center point coordinates and relative size of the target corresponding to each target detection box are input into a trained large language model, which outputs a final scene description statement corresponding to the pair of visible light and infrared images to be processed. This allows for the combination of target detection technology and scene description technology through LLM (Large Language Model) technology, which can overcome the shortcomings of previous image description models that are not applicable to low-light scenes or have inaccurate descriptions. In particular, it can generate accurate and detailed scene descriptions under insufficient light conditions, and has high application value.

[0117] To implement the above embodiments, a third aspect of the present invention provides an electronic device, comprising:

[0118] At least one processor; and a memory communicatively connected to said at least one processor;

[0119] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-described infrared-optical fusion scene semantic description method based on a large model.

[0120] To implement the above embodiments, a fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the above-described infrared-optical fusion scene semantic description method based on a large model.

[0121] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0122] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the infrared-optical fusion scene semantic description method based on a large model according to the above embodiments.

[0123] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0125] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0126] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0127] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0128] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0129] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0131] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A semantic description method for infrared-optical fusion scenes based on a large model, characterized in that, include: A visible light image and an infrared image from the same scene are acquired and matched. The matched visible light image and infrared image are then combined to form a visible light and infrared image pair to be processed. The visible light and infrared image pairs to be processed are input into a preset scene description model, and the scene description model outputs an initial scene description statement for the image pairs; The infrared image from the visible light and infrared image pair to be processed is input into a preset target detection model. The target detection model outputs the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box, thereby obtaining the center point coordinates and relative size of the target corresponding to each target detection box. The large language model is optimized by taking the initial scene description statement of the image pair, the type of target corresponding to each target detection box, the center point coordinates and relative size of the target corresponding to each target detection box, and the input prompts. The large language model outputs the final scene description statement corresponding to the visible light and infrared image pair to be processed.

2. The method according to claim 1, characterized in that, The scene description model adopts an RGB-IR multimodal fusion image description model.

3. The method according to claim 1, characterized in that, The target detection model used is the YOLOv7 target detection model.

4. The method according to claim 1, characterized in that, Before inputting the visible light and infrared images to be processed into a preset scene description model, the method further includes: Train the scene description model; Training the scene description model includes: 1) Using a paired visible light camera and an infrared camera, visible light images and corresponding infrared images of the same type of scene are acquired to obtain multiple sets of visible light and infrared image pairs, wherein the category of the scene is consistent with the category of the scene corresponding to the visible light and infrared image pair to be processed; 2) For each pair of visible light and infrared images obtained in step 1), scene description statements are annotated. The annotation content of the two images in each pair of visible light and infrared images is the same. The scene description statement reflects the target and its state in the scene in which the image is located. After the annotation is completed, each pair of visible light and infrared images and its corresponding scene description statement constitutes an image description training sample, and all image description training samples constitute the image description training set. 3) Use the image description training set obtained in step 2) to train the scene description model, and obtain the trained scene description model.

5. The method according to claim 4, characterized in that, Before inputting the infrared image from the pair of visible and infrared images to be processed into a preset target detection model, the method further includes: Train the target detection model; Training the target detection model includes: 1) Obtain an infrared image training set consisting of multiple labeled infrared images, wherein the scene category corresponding to each infrared image in the infrared image training set is consistent with the scene category corresponding to the image description training set, and each infrared image in the infrared image training set is labeled with the type of target corresponding to each target detection box in the image and the position information of the target detection box; 2) Train the target detection model using the infrared image training set obtained in step 1) to obtain the trained target detection model.

6. The method according to claim 5, characterized in that, The method further includes: 1) Input each pair of visible light and infrared images from the image description training set into the trained scene description model, and the scene description model outputs a scene description statement for the image pair; 2) Input each infrared image from the infrared image training set into the trained target detection model. The target detection model outputs the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box. The position information of the target detection box includes the coordinates of the upper left corner vertex and the lower right corner vertex of each target detection box. Then, calculate the center point coordinates and relative size of the target corresponding to each target detection box in each infrared image of the infrared image training set, as shown in the following expression: In the formula, x1 is the x-coordinate of the top left corner vertex of the target detection box, y1 is the y-coordinate of the top left corner vertex of the target detection box, x2 is the x-coordinate of the bottom right corner vertex of the target detection box, y2 is the y-coordinate of the bottom right corner vertex of the target detection box, x0 is the x-coordinate of the center point of the target detection box, y0 is the y-coordinate of the center point of the target detection box, and s is the relative size of the target. 3) Using the image description training set, the infrared image training set, and the results of steps 1) and 2), suggestions are made to optimize the large language model so that the large language model outputs a scene description statement that integrates multiple information corresponding to the input image scene.

7. The method according to claim 6, characterized in that, The suggestion to optimize the large language model includes: By combining the information output by the scene description model and the object detection model, a task description is designed for the large language model; Based on prior knowledge, rules are set for the large language model to meet user needs; Provide example text that conforms to the rules for the large language model.

8. A scene semantic description device based on a large model of infrared-optical fusion, characterized in that, include: The image acquisition module is used to acquire a visible light image and an infrared image of the same scene respectively and match them, and combine the matched visible light image and infrared image into a visible light and infrared image pair to be processed. An initial scene description module is used to input the visible light and infrared image pairs to be processed into a preset scene description model, and the scene description model outputs an initial scene description statement for the image pairs; The target detection module is used to input the infrared image from the visible light and infrared image pair to be processed into a preset target detection model. The target detection model outputs the type of target corresponding to each target detection box in the infrared image and the position information of the target detection box, thereby obtaining the center point coordinates and relative size of the target corresponding to each target detection box. The large language model description module is used to optimize the large language model by taking the initial scene description statement of the image pair, the type of the target corresponding to each target detection box, the center point coordinates and relative size of the target corresponding to each target detection box, and the large language model outputs the final scene description statement corresponding to the visible light and infrared image pair to be processed.

9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-7.

Images