Same-scene same-scale multi-modal image data acquisition method

Abstract

The invention discloses a same-scene same-scale multi-modal image data acquisition method, belongs to the technical field of multi-modal image data acquisition, and solves the problems that a multi-modal data set in the prior art is small in quantity scale, samples are difficult to align and target scales are different. Selecting an imaging scene, and placing the imaging target in the imaging scene; setting a target three-axis attitude angle and a multi-mode camera three-axis attitude angle; respectively setting camera exposure time according to the imaging scene and environment; adjusting the focal length of the camera to ensure that the image target scale is consistent with the scene size; adjusting the distance between the lens in the camera and the detector to ensure that the target is clearly imaged in the image; storing the same-scene same-scale multi-modal multi-frame images, and outputting a target three-axis attitude angle, a multi-modal camera three-axis attitude angle and a position coordinate of the target in a multi-modal camera coordinate system; according to the invention, the scale and richness of the multi-modal image data sample are improved; the multi-mode camera can work in the same view field, and it is guaranteed that multi-mode image data can be aligned and target scales are consistent.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal image data acquisition, and relates to a method for acquiring multimodal image data of the same scene and scale. Background Art

[0002] Each form or source of information can be called a modality. Multimodal images refer to single-modal images containing two or more light bands (for example, visible light, near infrared, mid-infrared, and far infrared). They are key input data for 3D adversarial sample generation and target detector training based on deep learning technology.

[0003] However, in the existing technology, such as the document "A Review of Multimodal Processing Technology under Data Constraints" (Wang Peijin, Yan Zhiyuan, Rong Xue'e, et al. Journal of Image and Graphics, 2022, 27(10): 2803-2834.), the multimodal datasets used have problems such as small size, difficult sample alignment, and different target scales. Summary of the invention

[0004] The technical solution of the present invention is used to solve the problems of small size, difficult sample alignment and inconsistent target scales in the prior art multimodal data sets.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] A method for acquiring multimodal image data of the same scene and the same scale comprises the following steps:

[0007] S1. Select an imaging scene and place an imaging target in the imaging scene; the imaging scene includes mountains, cities, and jungles; the imaging target includes but is not limited to pedestrians, vehicles, and animals;

[0008] S2, setting the target three-axis attitude angle and the multi-modal camera three-axis attitude angle; the target three-axis attitude angle includes the target azimuth angle, the target pitch angle, and the target roll angle; the multi-modal camera is composed of a visible light camera, a near-infrared camera, a mid-infrared camera, and a far-infrared camera; the multi-modal camera is installed and fixed on the intelligent gimbal to ensure that the visible light camera, the near-infrared camera, the mid-infrared camera, and the far-infrared camera image the same scene; the multi-modal camera three-axis attitude angle includes the multi-modal camera azimuth angle, the multi-modal camera pitch angle, and the multi-modal camera roll angle;

[0009] S3. According to the imaging scene and environment, the exposure time of the visible light camera, the near infrared camera, the mid infrared camera, and the far infrared camera are set respectively; the exposure time of the visible light camera and the near infrared camera can be set based on the imaging scene and the ambient light intensity level; the exposure time of the mid infrared camera and the far infrared camera can be set based on the imaging scene and the ambient temperature level;

[0010] S4, adjust the focal length of the visible light camera, near infrared camera, mid-infrared camera and far infrared camera to ensure that the target scale of the visible light, near infrared, mid-infrared and far infrared images is consistent with the scene size;

[0011] The relationship between the target scale and the camera focal length is as follows:

[0012]

[0013] The relationship between the scene size and the camera focal length is as follows:

[0014]

[0015] Among them, L object Indicates the target length; N pixel Indicates the number of pixels; S pixel Indicates the pixel size; D imaging Indicates imaging distance; D object represents the target scale; S scene Indicates the scene size;

[0016] S5. Adjust the distance between the lens and the detector in the visible light camera, near infrared camera, mid-infrared camera and far infrared camera to ensure that the target is clearly imaged in the visible light, near infrared, mid-infrared and far infrared images;

[0017] The adjustment range of the distance between the lens and the detector is as follows:

[0018]

[0019] Among them, F lens Indicates the focal ratio of the lens; D imaging Indicates imaging distance; C diameter represents the diameter of the circle of confusion;

[0020] S6. Save multiple frames of images of each modality in the same scene and scale, and output the target three-axis attitude angle, the multimodal camera three-axis attitude angle, and the target position coordinates (x, y, z) in the multimodal camera coordinate system. The multimodal camera coordinate system is a right-handed coordinate system, with the positive direction of the X axis pointing to the north and the positive direction of the Y axis pointing vertically upward.

[0021] A device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above-mentioned method for acquiring multi-modal image data of the same scene and the same scale, and the processor is configured to execute the program stored in the memory.

[0022] A storage medium stores a computer program, which, when executed by a processor, executes the steps of the above-mentioned method for acquiring multi-modal image data of the same scene and the same scale.

[0023] The advantages of the present invention are:

[0024] The technical solution of the present invention repeatedly acquires a large amount of multimodal image data under different imaging targets and scenes, thereby improving the sample size and richness of the multimodal image data; the multimodal camera composed of visible light, near-infrared, mid-infrared and far-infrared cameras can work in the same field of view, ensuring that the multimodal image data can be aligned and the target scale is consistent. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of the method for acquiring multi-modal image data of the same scene and the same scale of the present invention;

[0026] Figure 2 is a schematic diagram for explaining the scale of the method for acquiring multi-modal image data of the same scene and the same scale of the present invention;

[0027] Figure 3 This is a rendering of the experimental data of the method for acquiring multimodal image data of the same scene and the same scale of the present invention (the scale of the target in the scene is 0.2);

[0028] Figure 4 This is a rendering of the experimental data of the same-scene and same-scale multimodal image data acquisition method of the present invention (the scale of the target in the scene is 0.4). DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0031] Embodiment 1

[0032] like Figure 1 As shown, a method for acquiring multimodal image data of the same scene and scale includes the following steps:

[0033] Step 1: Select an imaging scene and place an imaging target in the imaging scene; the imaging scene includes mountains, cities, jungles, etc.; the imaging targets include but are not limited to pedestrians, vehicles, animals, etc.

[0034] Step 2, setting the target three-axis attitude angle and the multi-modal camera three-axis attitude angle; the target three-axis attitude angle includes the target azimuth angle, the target pitch angle, and the target roll angle; the multi-modal camera is composed of a visible light camera, a near-infrared camera, a mid-infrared camera, and a far-infrared camera; the multi-modal camera is installed and fixed on the intelligent gimbal to ensure that the visible light camera, the near-infrared camera, the mid-infrared camera, and the far-infrared camera image the same scene; the multi-modal camera three-axis attitude angle includes the multi-modal camera azimuth angle, the multi-modal camera pitch angle, and the multi-modal camera roll angle.

[0035] Step 3. Set the exposure time of the visible light camera, near-infrared camera, mid-infrared camera, and far-infrared camera respectively according to the imaging scene and environment; the exposure time of the visible light camera and near-infrared camera can be set based on the imaging scene and the ambient light intensity level; the exposure time of the mid-infrared camera and far-infrared camera can be set based on the imaging scene and the ambient temperature level.

[0036] Step 4: Adjust the focal lengths of the visible light camera, near infrared camera, mid-infrared camera, and far infrared camera to ensure that the target scales of the visible light, near infrared, mid-infrared, and far infrared images are consistent with the scene size;

[0037] The relationship between the target scale and the camera focal length is as follows:

[0038]

[0039] The relationship between the scene size and the camera focal length is as follows:

[0040]

[0041] Among them, L object Indicates the target length; N pixel Indicates the number of pixels; S pixel Indicates the pixel size; D imaging Indicates imaging distance; D object represents the target scale; S scene Indicates the size of the scene.

[0042] Step 5: Adjust the distance between the lens and the detector in the visible light camera, near infrared camera, mid-infrared camera and far infrared camera to ensure that the target is clearly imaged in the visible light, near infrared, mid-infrared and far infrared images;

[0043] The adjustment range of the distance between the lens and the detector is as follows:

[0044]

[0045] Among them, F lens Indicates the focal ratio of the lens; D imaging Indicates imaging distance; Cdiameter Represents the diameter of the circle of confusion.

[0046] Step 6: Save multiple frames of images of each modality in the same scene and scale, and output the target three-axis attitude angle, the multimodal camera three-axis attitude angle, and the target position coordinates (x, y, z) in the multimodal camera coordinate system. The multimodal camera coordinate system is a right-handed coordinate system, with the positive direction of the X axis pointing to due north and the positive direction of the Y axis pointing vertically upward.

[0047] like Figure 2 As shown, in this embodiment, the scale of the target in the scene=A1 / A2, where the units of A1 and A2 are pixels.

[0048] like Figure 3 As shown, the multimodal image data of the same scene and scale in this embodiment includes a visible light image and a mid-infrared image, and the scale of the target in the scene is 0.2. Figure 4 As shown, the multimodal image data of the same scene and scale in this embodiment includes a visible light image and a mid-infrared image, and the scale of the target in the scene is 0.4. Figure 3 and Figure 4 It can be seen that the method of the present invention can repeatedly acquire a large amount of multimodal image data under different imaging targets and scenes, thereby improving the sample size and richness of multimodal image data; the multimodal camera composed of visible light, near-infrared, mid-infrared and far-infrared cameras can work in the same field of view, ensuring that the multimodal image data can be aligned and the target scale is consistent.

[0049] Embodiment 2

[0050] A device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the method for acquiring multi-modal image data of the same scene and scale in embodiment 1, and the processor is configured to execute the program stored in the memory.

[0051] Embodiment 3

[0052] A storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for acquiring multi-modal image data of the same scene and the same scale in embodiment 1 are executed.

[0053] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for acquiring multimodal image data of the same scene and scale, characterized in that: The following steps are involved: S1, select an imaging scene and place the imaging target in the imaging scene; S2, setting the target three-axis attitude angle and the multi-modal camera three-axis attitude angle; the target three-axis attitude angle includes the target azimuth angle, the target pitch angle, and the target roll angle; the multi-modal camera is composed of a visible light camera, a near-infrared camera, a mid-infrared camera, and a far-infrared camera; the multi-modal camera is installed and fixed on the intelligent gimbal to ensure that the visible light camera, the near-infrared camera, the mid-infrared camera, and the far-infrared camera image the same scene; the multi-modal camera three-axis attitude angle includes the multi-modal camera azimuth angle, the multi-modal camera pitch angle, and the multi-modal camera roll angle; S3. According to the imaging scene and environment, the exposure time of the visible light camera, the near infrared camera, the mid infrared camera, and the far infrared camera are set respectively; the exposure time of the visible light camera and the near infrared camera can be set based on the imaging scene and the ambient light intensity level; the exposure time of the mid infrared camera and the far infrared camera can be set based on the imaging scene and the ambient temperature level; S4, adjust the focal length of the visible light camera, near infrared camera, mid-infrared camera and far infrared camera to ensure that the target scale of the visible light, near infrared, mid-infrared and far infrared images is consistent with the scene size; The relationship between the target scale and the camera focal length is as follows: The relationship between the scene size and the camera focal length is as follows: Among them, L object Indicates the target length; N pixel Indicates the number of pixels; S pixel Indicates the pixel size; D imaging Indicates imaging distance; D object represents the target scale; S scene Indicates the scene size; S5. Adjust the distance between the lens and the detector in the visible light camera, near infrared camera, mid-infrared camera and far infrared camera to ensure that the target is clearly imaged in the visible light, near infrared, mid-infrared and far infrared images; The adjustment range of the distance between the lens and the detector is as follows: Among them, F lens Indicates the focal ratio of the lens; D imaging Indicates imaging distance; C diameter represents the diameter of the circle of confusion; S6. Save multiple frames of images of each modality in the same scene and scale, and output the target three-axis attitude angle, the multimodal camera three-axis attitude angle, and the target position coordinates (x, y, z) in the multimodal camera coordinate system. The multimodal camera coordinate system is a right-handed coordinate system, with the positive direction of the X axis pointing to the north and the positive direction of the Y axis pointing vertically upward.

2. The method for acquiring multi-modal image data of the same scene and scale according to claim 1, characterized in that: The imaging scenes described in step S1 include mountains, cities, and jungles.

3. The method for acquiring multi-modal image data of the same scene and scale according to claim 1, characterized in that: The imaging targets described in step S1 include but are not limited to pedestrians, vehicles, and animals.

4. A device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the method for acquiring multi-modal image data of the same scene and scale as described in any one of claims 1 to 3, and the processor is configured to execute the program stored in the memory.

5. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for acquiring multi-modal image data of the same scene and scale as claimed in any one of claims 1 to 3 are executed.

Images