Multi-modal small target detection method and device and electronic equipment

Abstract

The invention provides a multi-mode small target detection method and device and electronic equipment. According to the method, before the multi-modal model is passed, the deployed target detection model is utilized to perform target detection on the to-be-detected image, and for the target object which meets the small target requirement and is included in the to-be-detected image, the target object area corresponding to the target object is obtained according to the image input requirement of the multi-modal model and the target object; the small target needing to be detected in the to-be-detected image is independently extracted, the target object area corresponding to the target object is determined according to the image input requirement of the multi-modal model, and the target object area and the to-be-detected image are input into the modal model, so that when the multi-modal model is used for detecting the small target, the detection accuracy is improved, and the detection efficiency is improved. According to the invention, the detail features of the small target can be better obtained through the target object region, the condition of misrecognition is avoided, and the detection accuracy of the multi-modal model to the small target is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multimodal small target detection method, device and electronic equipment. Background Art

[0002] In real-world applications, multimodal models are often used to detect images and describe the behavior of specific types of objects within them. For example, a multimodal model can generate a specific description of a pedestrian, such as "a pedestrian in a red coat walking on an empty street," by feeding it an image containing a pedestrian. However, in such scenarios, if the pedestrian occupies a relatively small portion of the image, the multimodal model's description of the pedestrian's behavior may be biased or even inaccurate. Summary of the Invention

[0003] In view of this, the present application provides a multimodal small target detection method, device and electronic device to improve the detection accuracy of the multimodal model for small targets.

[0004] The technical solutions provided in this application are as follows:

[0005] According to an embodiment of the first aspect of the present application, a multimodal small target detection method is provided, the method comprising:

[0006] Performing target detection on the image to be tested using the deployed target detection model to obtain location information of the target object in the image to be tested; the target object corresponds to the target category involved in detection using the current multimodal model;

[0007] Based on the position information of the target object in the image to be measured and the image size information of the image to be measured, determining whether the target object is an object that meets the small target requirement; if so, obtaining a target object area according to the image input requirements of the multimodal model and the target object in the image to be measured; the target object area refers to an area that at least includes the target object;

[0008] The image to be measured and the target object region are input into the multimodal model to obtain description information about the target object.

[0009] Optionally, performing target detection on the image to be tested using the deployed target detection model to obtain position information of the target object in the image to be tested includes:

[0010] Performing target detection on the image to be tested using the deployed target detection model to obtain object position information; the object position information includes position information of at least one object in the image to be tested in the image to be tested;

[0011] Based on the target category currently requiring multimodal model detection, position information of the target object corresponding to the target category in the image to be detected is determined from the obtained object position information.

[0012] Optionally, determining whether the target object meets the small target requirement based on position information of the target object in the image to be measured and image size information of the image to be measured includes:

[0013] Determining the occupied size information of the target object in the image to be measured based on the position information of the target object in the image to be measured;

[0014] It is determined whether the target object meets the small target requirement according to the occupied size information and the image size information of the image to be measured.

[0015] Optionally, the image input requirement of the multimodal model includes an image aspect ratio requirement; and obtaining the target object area according to the image input requirement of the multimodal model and the target object in the image to be measured includes:

[0016] Selecting a reference area from the image to be measured; the reference area includes the target object and a background image around the target object;

[0017] Detecting whether the aspect ratio of the reference area matches the image aspect ratio requirement;

[0018] If the aspect ratio of the reference area does not match the image aspect ratio requirement, a target object area including the reference area is selected from the image to be measured; the aspect ratio of the target object area matches the image aspect ratio requirement.

[0019] Optionally, the image input requirement of the multimodal model includes an image resolution requirement; and inputting the image to be measured and the target object area into the multimodal model to obtain descriptive information about the target object includes:

[0020] Detecting whether the target object area and the resolution of the image to be measured match the image resolution requirement;

[0021] If the resolution of the target object area and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object area and / or the image to be measured so that the adjusted resolution of the target object area and the image to be measured match the image resolution requirement;

[0022] The adjusted target object region is used as a prompt image of the adjusted image to be tested, and the adjusted target object region and the adjusted image to be tested are detected based on the multimodal model to obtain description information about the target object.

[0023] Optionally, the image input requirement of the multimodal model includes an image resolution requirement; and inputting the image to be measured and the target object area into the multimodal model to obtain descriptive information about the target object includes:

[0024] Adjusting the resolution of the target object area and / or the image to be measured so that the adjusted target object area and the image to be measured meet the image stitching requirements;

[0025] Stitching the target object area that meets the image stitching requirements and the image to be tested to obtain a stitched image;

[0026] Detecting whether the resolution of the stitched image matches the image resolution requirement;

[0027] If the resolution of the target object area and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object area and / or the image to be measured so that the adjusted resolution of the target object area and the image to be measured match the image resolution requirement;

[0028] The spliced image is input into the multimodal model to detect the spliced image based on the multimodal model to obtain description information about the target object.

[0029] Optionally, the image input requirement of the multimodal model includes an image resolution requirement; and the method further includes:

[0030] If it is determined based on the position information of the target object in the image to be measured and the image size information of the image to be measured that the target object does not meet the small target requirement, detecting whether the resolution of the image to be measured matches the image resolution requirement;

[0031] If not, adjusting the resolution of the image to be tested to obtain an image to be input, wherein the resolution of the image to be input matches the image resolution requirement;

[0032] The image to be input is input into the multimodal model to obtain description information about the target object that does not meet the small target requirement.

[0033] According to an embodiment of the second aspect of the present application, a multimodal small target detection device is provided, the device comprising:

[0034] A detection unit, configured to perform target detection on the image to be tested using the deployed target detection model to obtain location information of the target object in the image to be tested; the target object corresponds to the target category involved in detection using the current multimodal model;

[0035] a determining unit, configured to determine whether the target object satisfies the small target requirement based on position information of the target object in the image to be measured and image size information of the image to be measured, and if so, obtain a target object region according to the image input requirements of the multimodal model and the target object in the image to be measured; the target object region refers to a region that at least includes the target object;

[0036] An inference unit is used to input the image to be measured and the target object area into the multimodal model to obtain description information about the target object.

[0037] Optionally, the detection unit is specifically used to:

[0038] Performing target detection on the image to be tested using the deployed target detection model to obtain object position information; the object position information includes position information of at least one object in the image to be tested in the image to be tested;

[0039] Based on the target category currently requiring multimodal model detection, determining, from the obtained object position information, position information of a target object corresponding to the target category in the image to be tested;

[0040] And / or, the determining unit is specifically configured to:

[0041] Determining the occupied size information of the target object in the image to be measured based on the position information of the target object in the image to be measured;

[0042] determining whether the target object meets the small target requirement according to the occupied size information and the image size information of the image to be measured;

[0043] And / or, the image input requirement of the multimodal model includes an image aspect ratio requirement; and the determining unit is specifically configured to:

[0044] Selecting a reference area from the image to be measured; the reference area includes the target object and a background image around the target object;

[0045] Detecting whether the aspect ratio of the reference area matches the image aspect ratio requirement;

[0046] If the aspect ratio of the reference area does not match the image aspect ratio requirement, selecting a target object area including the reference area from the image to be measured; the aspect ratio of the target object area matches the image aspect ratio requirement;

[0047] And / or, the image input requirement of the multimodal model includes an image resolution requirement; and the inference unit is specifically configured to:

[0048] Detecting whether the target object area and the resolution of the image to be measured match the image resolution requirement;

[0049] If the resolution of the target object area and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object area and / or the image to be measured so that the adjusted resolution of the target object area and the image to be measured match the image resolution requirement;

[0050] Using the adjusted target object region as a prompt image of the adjusted image to be tested, and detecting the adjusted target object region and the adjusted image to be tested based on the multimodal model to obtain description information about the target object;

[0051] And / or, the image input requirement of the multimodal model includes an image resolution requirement; and the inference unit is specifically configured to:

[0052] Adjusting the resolution of the target object area and / or the image to be measured so that the adjusted target object area and the image to be measured meet the image stitching requirements;

[0053] Stitching the target object area that meets the image stitching requirements and the image to be tested to obtain a stitched image;

[0054] Detecting whether the resolution of the stitched image matches the image resolution requirement;

[0055] If the resolution of the target object area and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object area and / or the image to be measured so that the adjusted resolution of the target object area and the image to be measured match the image resolution requirement;

[0056] Inputting the spliced image into the multimodal model to detect the spliced image based on the multimodal model to obtain description information about the target object;

[0057] And / or, the image input requirement of the multimodal model includes an image resolution requirement; and the reasoning unit is further configured to:

[0058] If it is determined based on the position information of the target object in the image to be measured and the image size information of the image to be measured that the target object does not meet the small target requirement, detecting whether the resolution of the image to be measured matches the image resolution requirement;

[0059] If not, adjusting the resolution of the image to be tested to obtain an image to be input, wherein the resolution of the image to be input matches the image resolution requirement;

[0060] The image to be input is input into the multimodal model to obtain description information about the target object that does not meet the small target requirement.

[0061] According to an embodiment of the third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0062] It can be seen from the above technical solution that before the multimodal model is used, the present application first uses the deployed target detection model to perform target detection on the image to be tested. When it is determined that the image to be tested includes a target object that meets the small target requirements, the target object area corresponding to the target object is obtained according to the image input requirements of the multimodal model and the target object, and the image to be tested and the target object area are further input into the multimodal model to obtain description information about the target object. Through the solution proposed in this application, the small target that needs to be detected in the image to be tested is extracted separately, and the target object area corresponding to the target object is determined according to the image input requirements of the multimodal model, and the target object area and the image to be tested are both input into the modal model, so that when the multimodal model detects the small target, it can better obtain the detailed features of the small target through the target object area, avoid misidentification, and improve the detection accuracy of the multimodal model for small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0064] Figure 1 A schematic diagram of a multimodal small model detection scenario provided in an embodiment of the present application;

[0065] Figure 2 A flowchart of a multimodal small target detection method according to an embodiment of the present application;

[0066] Figure 3 A schematic diagram of a specific process flow of a multimodal small target detection method provided in an embodiment of the present application;

[0067] Figure 4 Schematic diagram of the execution framework of the multimodal small target detection method provided in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0069] Figure 6 This is a structural diagram of the multimodal small target detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0071] In actual application scenarios, multimodal models are often used to detect images to describe the behavior of objects of specified types in the images.

[0072] Please refer to Figure 1 , Figure 1 Schematic diagram of a multimodal small model detection scenario provided in an embodiment of the present application.

[0073] like Figure 1 As shown in Figure 2, the process of detecting the image to be tested using the multimodal model is as follows:

[0074] An image to be tested that includes a pedestrian is adjusted to an input image size supported by the multimodal model, and the adjusted image is input into the multimodal model to obtain descriptive information of the specified object in the image to be tested. For example, if the prompt text pre-set for the multimodal model is "Describe the behavior of the pedestrian in the image to be tested", then after inputting the above image to be tested into the multimodal model, a description of the pedestrian in the image can be obtained, such as "In the early morning, a person riding an electric bike crosses the empty street."

[0075] However, if the pedestrian occupies a small proportion in the image to be tested, the multimodal model will have deviations or even errors in describing the pedestrian's behavior. Figure 1 In the image under test, it can be seen that the pedestrian is not actually riding an electric scooter, but walking on the road. Because the pedestrian occupies a small proportion of the image under test and is far away from the image acquisition location, the pedestrian's clothing causes visual errors, making the pedestrian appear to be riding an electric scooter in the overall image under test. In this case, the multimodal model actually misrepresents the small target (pedestrian) in the image under test.

[0076] Based on this, this application proposes a multimodal small target detection method to avoid the misidentification of small targets by the multimodal model and improve the detection accuracy of the multimodal model for small targets.

[0077] Please refer to Figure 2 , Figure 2 This is a flowchart of the multimodal small target detection method provided in an embodiment of the present application.

[0078] like Figure 2 As shown, the method may include the following steps:

[0079] Step 201 : Utilize the deployed target detection model to perform target detection on the image to be tested, and obtain the position information of the target object in the image to be tested.

[0080] The target object corresponds to the target category involved in detection using the current multimodal model.

[0081] In this embodiment, before inputting the image to be tested into the multimodal model, the image to be tested may be first input into a deployed target detection model for target detection to obtain position information of the target object in the image to be tested.

[0082] In this embodiment, the target detection model can be a convolutional neural network region candidate detection model (R-CNN: Regions with Convolutional Neural Networks), a fast convolutional neural network region candidate detection model (Fast R-CNN: Fast Regions with Convolutional Neural Networks) or "You Only Look Once" (YOLO: You Only Look Once), etc., and this application does not limit this.

[0083] As an embodiment, a specific method for performing target detection on an image to be tested using a deployed target detection model and obtaining position information of a target object in the image to be tested may include:

[0084] Using the deployed object detection model to perform object detection on the image to be tested, and obtain object position information; the object position information includes position information of at least one object in the image to be tested;

[0085] Based on the target category currently requiring multimodal model detection, the position information of the target object corresponding to the target category in the image to be tested is determined from the obtained object position information.

[0086] Generally speaking, the target detection model will detect objects of multiple categories included in the image to be tested, mark the area to which each object belongs in the image to be tested (i.e., the object location information of the object) through a position box, and mark the category corresponding to the object.

[0087] For example, for Figure 1 After inputting the image to be tested into the target detection model, categories such as pedestrians, vehicles, trees, and buildings can be obtained, and each category includes one or more objects.

[0088] In this embodiment, the multimodal model is pre-configured with the target categories that it needs to detect (i.e., the target categories involved when detecting using the current multimodal model). The specific method of configuring the target categories that need to be detected can be to pre-specify text prompt information for the multimodal model. For example, if the user wants the multimodal model to detect the behavior of objects classified as "pedestrians" in the input image, the multimodal model can be pre-configured with text prompt information such as "Please describe the behavior of pedestrians included in the image to be tested" so that when the multimodal model receives the input image, it can detect the pedestrian area in the input image, further determine the behavior of pedestrians in the area and describe it.

[0089] After obtaining the position information of each object in the image to be tested, the position information of the target object corresponding to the target category can be determined from the obtained object position information based on the target category to be detected in the current multimodal model. In other words, the target object matching the target category is determined from the multiple objects in the image to be tested, and this target object is selected as the object to be subsequently processed.

[0090] In this embodiment, the position information of the target object corresponding to the target category in the image to be tested can be a rectangular area corresponding to the target object in the image to be tested, as detected by the target detection model. The rectangular area can be represented by the coordinate positions of the four vertices of the rectangular area, or can be represented by the coordinates of any vertex of the rectangular area, as well as the width and height of the rectangular area, and this application is not limited to this.

[0091] At this point, the description of step 201 ends, and step 202 is executed next.

[0092] Step 202: Based on the position information of the target object in the image to be tested and the image size information of the image to be tested, determine whether the target object meets the small target requirement. If so, obtain the target object area according to the image input requirements of the multimodal model and the target object in the image to be tested.

[0093] The target object area refers to an area that at least includes the target object.

[0094] In this embodiment, for the target object that needs to be identified, if the target object is a small target, it is easy for the detailed features to be unclear, resulting in recognition deviation or even error. Therefore, after determining the target object that needs to be identified in the image to be tested through step 201, it can be further determined whether the target object is a small target.

[0095] In this embodiment, whether the target object meets the small target requirement may be determined based on the position information of the target object in the image to be measured and the image size information of the image to be measured.

[0096] Specifically, the occupied size information of the target object in the image to be measured can be determined based on the position information of the target object in the image to be measured; and whether the target object meets the small target requirement can be determined based on the occupied size information and the image size information of the image to be measured.

[0097] As an embodiment, the size information occupied by the target object in the image to be measured may be the width value and height value of the target object. Based on the position information of the target object in the image to be measured, such as the coordinate positions of the four vertices of the rectangular frame area corresponding to the target object, the width value and height value of the rectangular frame area corresponding to the target object may be obtained, and the width value and height value may be used as the width value and height value of the target object.

[0098] In the case where the size information occupied by the target object in the image to be measured is the width and height of the target object, a specific method for determining whether the target object meets the small target requirement based on the size information occupied and the image size information of the image to be measured may include:

[0099] If the ratio of the width value of the target object to the width value of the image to be measured is less than the first threshold, and the ratio of the height value of the target object to the height value of the image to be measured is less than the second threshold, the target object is determined to be an object that meets the small target requirement.

[0100] For example, if the width value of the target object is less than 1 / 10 of the width value of the image to be measured (that is, the ratio of the width value of the target object to the width value of the image to be measured is less than the first threshold value of 1 / 10), and the height value of the target object is less than 1 / 10 of the height value of the image to be measured (that is, the ratio of the width value of the target object to the width value of the image to be measured is less than the second threshold value of 1 / 10), the target object is determined to be an object that meets the requirements of a small target. At this time, the target object can be regarded as a small target, wherein the above-mentioned first threshold value and the second threshold value can be adjusted according to the sensitivity of the specific task.

[0101] As an embodiment, the size information occupied by the target object in the image to be measured can also be the area of the region where the target object is located. According to the position information of the target object in the image to be measured, such as the coordinate positions of the four vertices of the rectangular frame area corresponding to the target object, the width value and height value of the rectangular frame area corresponding to the target object can be obtained, and the area of the region where the target object is located can be determined based on the width value and the height value.

[0102] In the case where the occupied size information of the target object in the image to be measured is the area of the region where the target object is located, a specific method for determining whether the target object meets the small target requirement based on the occupied size information and the image size information of the image to be measured may include:

[0103] If the ratio of the area of the region where the target object is located to the area of the image to be measured is less than a third threshold, the target object is determined to be an object that meets the small target requirement.

[0104] For example, if the area of the region where the target object is located is smaller than 1 / 20 of the area of the image to be measured (i.e., the ratio of the area of the region where the target object is located to the area of the image to be measured is smaller than the third threshold value 1 / 20), the target object is determined to be an object that meets the requirements of a small target. At this time, the target object can be regarded as a small target, wherein the above-mentioned third threshold value can also be adjusted according to the sensitivity of the specific task.

[0105] After determining whether the target object is a small target through the above method, if the target object is determined to be an object that meets the small target requirements, the target object area can be obtained according to the image input requirements of the multimodal model and the target object in the image to be tested.

[0106] In this embodiment, when it is determined that the target object is a small target, it indicates that the image to be tested is directly input into the multimodal model for detection. Since the target object is small, its detailed features are not clear enough, and the description information obtained for the target object may be biased. Therefore, the target object area corresponding to the target object can be determined, and the target object area can be used as a prompt image of the image to be tested to provide more detailed features.

[0107] Exemplarily, the image input requirements of the multimodal model include image aspect ratio requirements. Based on the image input requirements of the multimodal model and the target object in the image to be measured, a specific method for obtaining the target object area may include:

[0108] A reference area is selected from the image to be tested; the reference area includes a target object and a background image surrounding the target object; and an aspect ratio of the reference area is detected to determine whether it matches an image aspect ratio requirement; if the aspect ratio of the reference area does not match the image aspect ratio requirement, a target object area including the reference area is selected from the image to be tested; and the aspect ratio of the target object area matches the image aspect ratio requirement.

[0109] In this embodiment, a region including the target object and the background image surrounding the target object may be selected from the image to be tested as a reference region. A specific method for determining the reference region may be to expand the target object by a specified proportion based on the position information of the target object to obtain the reference region.

[0110] Considering that the background image around the target object helps to provide some related information about the target object, for example, if the target object is holding a badminton racket, it indicates that the target object may be playing badminton; for example, if there is a bicycle under the target object, it indicates that the target object may be riding a bicycle, in this embodiment, the target object can be expanded at a specified proportion based on the position information of the target object, such as expanding the width and height of the rectangular frame area corresponding to the target object by 1.1 times to obtain a reference area to include some background information around the target object, wherein the specified expansion ratio can also be adjusted according to actual needs, and this application does not limit this.

[0111] It is easy to understand that compared with the target object, the reference area includes the background image around the target object. Since the background image around the target object may include tools or related items held by the target object, it is more conducive to analyzing the behavior of the target object.

[0112] After obtaining the reference area corresponding to the target object, it is further considered that when the area is input into the multimodal model as a prompt image, it is necessary to input it according to the image size specified by the image input requirements of the multimodal model. The image input requirements of the multimodal model include image aspect ratio requirements, that is, the aspect ratio of the input image supported by the multimodal model (that is, the width-to-height ratio of the image) is fixed. For example, the aspect ratio of the input image is required to be fixed at 1:1, that is, the image is input in the form of a square. At this time, it is possible to first detect whether the aspect ratio of the determined reference area meets the image aspect ratio requirements. If so, the reference area can be directly used as the target object area corresponding to the target object.

[0113] If not, a target object region including a reference region may be selected from the image to be measured according to the image aspect ratio requirement, and the aspect ratio of the target object region matches the image aspect ratio requirement.

[0114] Exemplarily, the method of selecting the target object area including the reference image area from the image to be tested according to the image aspect ratio requirement may be to fix the width of the reference area and adjust the height of the reference area so that the aspect ratio (i.e., the width-to-height ratio) of the adjusted reference area meets the image aspect ratio requirement; or to fix the height of the reference area and adjust the width of the reference area so that the aspect ratio (i.e., the width-to-height ratio) of the adjusted reference area meets the image aspect ratio requirement; or to use a vertex of the reference area as a starting point and delineate the target object area along the width and height directions of the reference area according to the image aspect ratio requirement until the target object area includes the reference area. The present application does not limit the method for determining the target object area. It only needs to ensure that the determined target object area completely includes the reference area. That is, in the process of determining the target object area by the above method, if the target object area does not include the reference area, other methods can be used to determine the target object area to ensure that the target object area completely includes the reference area.

[0115] At this point, the description of step 202 ends, and step 203 is executed next.

[0116] Step 203: Input the image to be tested and the target object region into the multimodal model to obtain description information about the target object.

[0117] In this embodiment, after the target object region is determined, the image to be tested and the target object region may be detected based on the multimodal model to obtain description information about the target object in the target object region.

[0118] Specifically, inputting the image to be measured and the target object area into the multimodal model may include two solutions.

[0119] In solution 1, the multimodal model processes two separate images.

[0120] As an embodiment, the image input requirements of the multimodal model include image resolution requirements. The specific method of inputting the image to be measured and the target object area into the multimodal model to obtain description information about the target object may include:

[0121] Detect whether the resolution of the target object area and the image to be tested matches the image resolution requirement; if the resolution of the target object area and / or the image to be tested does not match the image resolution requirement, adjust the resolution of the target object area and / or the image to be tested so that the adjusted resolution of the target object area and the image to be tested match the image resolution requirement; use the adjusted target object area as a prompt image for the adjusted image to be tested, and detect the adjusted target object area and the adjusted image to be tested based on the multimodal model to obtain descriptive information about the target object.

[0122] Specifically, the image input requirements configured for the multimodal model may also include image resolution requirements, meaning that the multimodal model only supports image inputs that meet the image resolution requirements. Based on this, it is possible to first detect whether the resolution of the target object area and the image to be tested matches the image resolution requirements. If both the target object area and the image to be tested do match the image resolution requirements, then both the target object area and the image to be tested meet the input requirements of the multimodal model. In this case, the target object area can be used as a prompt image for the image to be tested, and the adjusted target object area and the adjusted image to be tested can be detected based on the multimodal model to obtain descriptive information about the target object.

[0123] If the resolution of the target object region and / or the image to be tested does not match the required image resolution, it indicates that at least one image does not meet the input requirements of the multimodal model. In this case, the resolution of the target object region and / or the image to be tested can be adjusted so that the adjusted resolution of the target object region and the image to be tested match the required image resolution. The adjusted target object region is further used as a prompt image for the adjusted image to be tested. The adjusted target object region and the adjusted image to be tested are detected based on the multimodal model to obtain descriptive information about the target object.

[0124] As an embodiment, the target object area and the image to be tested can be annotated, such as by annotating text information, marking the target object area as a prompt image and the image to be tested as an image to be detected, so that the multimodal model can know which image is the prompt image and which image is the image to be detected when processing the two images.

[0125] As an embodiment, the target object area and the image to be tested may not be labeled, but may be input into the multimodal model in a fixed order. For example, each input is input first, and then the target object area, so that the multimodal model can know which image is the prompt image and which image is the image to be tested.

[0126] In this embodiment, in the above-mentioned solution 1, the two input images can be directly inferred in a multi-batch manner based on the multimodal model to obtain accurate description information of the target object.

[0127] In the second solution, the multimodal model processes an image that is composed of the target object area and the image to be tested.

[0128] As an embodiment, the image input requirements of the multimodal model include image resolution requirements; the image to be measured and the target object area are input into the multimodal model to obtain description information about the target object, including:

[0129] Adjust the resolution of the target object area and / or the image to be tested so that the adjusted target object area and the image to be tested meet the image stitching requirements; stitch the target object area and the image to be tested that meet the image stitching requirements to obtain a stitched image; detect whether the resolution of the stitched image matches the image resolution requirements; if the resolution of the target object area and / or the image to be tested does not match the image resolution requirements, adjust the resolution of the target object area and / or the image to be tested so that the adjusted target object area and the image to be tested match the image resolution requirements; input the stitched image into a multimodal model to detect the stitched image based on the multimodal model to obtain descriptive information about the target object.

[0130] Specifically, since this solution is to stitch the target object area and the image to be tested into one image and then process it through a multimodal model, the resolution of the target object area and / or the image to be tested can be adjusted first so that the adjusted target object area and the image to be tested meet the image stitching requirements, that is, the resolution of the target object area and the image to be tested can be adjusted to the same resolution for easy stitching.

[0131] Furthermore, the target object region and the image to be tested that meet the image stitching requirements can be stitched together to obtain a stitched image. For example, the target object region and the image to be tested can be stitched together left and right to obtain a stitched image, or the target object region and the image to be tested can be stitched together top and bottom to obtain a stitched image, and this application does not limit this.

[0132] After completing image stitching and obtaining the stitched image, we can first check whether the resolution of the stitched image matches the image resolution requirement. If it matches, it means that the stitched image meets the input requirements of the multimodal model. At this time, the stitched image can be directly detected based on the multimodal model to obtain descriptive information about the target object.

[0133] If there is a mismatch, it indicates that the stitched image does not meet the input requirements of the multimodal model. In this case, the resolution of the stitched image can be adjusted to match the image resolution requirements. The adjusted stitched image is then tested based on the multimodal model to obtain a description of the target object.

[0134] This concludes the description of the above two solutions.

[0135] It is easy to understand that no matter which of the above two solutions is used, when the multimodal model detects the small target (target object), it can better obtain the detailed features of the small target through the target object area, and can also describe the characteristics of the target object more accurately.

[0136] This concludes the description of step 203.

[0137] As an embodiment, the above describes a processing method for determining that the target object is an object that meets the small target requirements. If, based on the position information of the target object in the image to be measured and the image size information of the image to be measured, it is determined that the target object is not an object that meets the small target requirements, then it is detected whether the resolution of the image to be measured matches the image resolution requirements; if not, the resolution of the image to be measured is adjusted to obtain the image to be input, and the resolution of the image to be input matches the image resolution requirements; the image to be input is input into the multimodal model to obtain descriptive information about the target object that does not meet the small target requirements.

[0138] In this embodiment, if it is determined that the target object does not meet the small target requirements, the image to be tested can be directly adjusted to match the image resolution requirements, and the input model can be directly input into the multimodal model to obtain descriptive information about the target object that does not meet the small target requirements.

[0139] Since the target object is not a small target, it can directly obtain relatively rich detail features in the image to be tested. The image to be tested can be directly inferred and detected based on the multimodal model to obtain descriptive information about the target object that does not meet the small target requirements.

[0140] This concludes Figure 2 Description of the multimodal small object detection method in

[15] .

[0141] Before using the multimodal model, the present application first uses the deployed target detection model to perform target detection on the image to be tested. When it is determined that the image to be tested includes a target object that meets the small target requirements, the target object area corresponding to the target object is obtained according to the image input requirements of the multimodal model and the target object. The image to be tested and the target object area are further input into the multimodal model to obtain description information about the target object. Through the scheme proposed in the present application, the small target that needs to be detected in the image to be tested is extracted separately, and the target object area corresponding to the target object is determined according to the image input requirements of the multimodal model. The target object area and the image to be tested are both input into the modal model, so that when the multimodal model detects the small target, it can better obtain the detailed features of the small target through the target object area, avoid misidentification, and improve the detection accuracy of the multimodal model for small targets.

[0142] Below through Figure 3 as well as Figure 4 The multimodal small target detection method proposed in this application is described.

[0143] Please refer to Figure 3 , Figure 3A schematic diagram of the specific process of the multimodal small target detection method provided in the embodiment of the present application.

[0144] like Figure 3 As shown, in this embodiment, the method as a whole may include the following specific steps:

[0145] The image to be tested is input into the trained target detection model. If the target detection model determines that the target object to be detected in the image to be tested is a small target, a reference area corresponding to the small target is determined from the image to be tested. The reference area includes the small target and the background area around the small target. Further, according to the input requirements of the multimodal model, the target object area whose aspect ratio meets the image aspect ratio requirements is determined. The target object area includes the above-mentioned reference area.

[0146] Furthermore, the target object region and the image to be tested are spliced together to obtain a spliced image, which is then fed into a multimodal model for detection to obtain a description of the small target. Alternatively, the target object region and the image to be tested are directly fed into a multimodal model for detection to obtain a description of the small target. (The step of adjusting the resolution to meet the image resolution requirements of the multimodal model is omitted here.)

[0147] If the target detection model determines that the target object to be detected in the image to be tested is not a small target, the image to be tested can be directly adjusted to a size supported by the multimodal model and then input into the multimodal model for detection.

[0148] This concludes Figure 3 Description of the specific flow chart of the multimodal small target detection method.

[0149] Please refer to Figure 4 , Figure 4 Schematic diagram of the execution framework of the multimodal small target detection method provided in an embodiment of the present application.

[0150] like Figure 4 As shown, the solution proposed by this application is Figure 1 The same test image is detected. First, the test image is input into the target detection model, and the target object (that is, the area to which the pedestrian belongs) is determined to be an object that meets the small target requirements. Then, the reference area corresponding to the target object is further determined (for example, the area to which the target object belongs is expanded to obtain the image of the pedestrian and its surrounding background, as shown in the figure).

[0151] Next, based on the image aspect ratio requirements supported by the multimodal model, the target object area can be obtained from the image to be tested. The target object area includes the above-mentioned reference area. Specifically, the above-mentioned reference area can be adjusted according to the image aspect ratio requirements supported by the multimodal model to obtain the target object area to the right of "Determine the target object area" in the figure.

[0152] Next, select one of the two methods: image stitching or separate input, and process the target object area and the image to be measured based on the multimodal model. (The step of adjusting the resolution to meet the image resolution requirements of the multimodal model is omitted here)

[0153] At this time, since the multimodal model can use the target object area as a prompt image when identifying pedestrians, it can better obtain the detailed features of the small target through the target object area, so it can accurately identify that the pedestrian is walking rather than riding an electric bike, and then output a description of the pedestrian's behavior: In the early morning, a person in a red coat is walking on an empty road.

[0154] It is easy to understand that the solution proposed in this application can take into account the macro features of the entire image while better understanding the detailed features of small targets, thereby improving the detection accuracy of the multimodal model for small targets.

[0155] If the target detection model determines that the target object (i.e., the area to which the pedestrian belongs) does not meet the small target requirements, the resolution of the image to be tested can be directly adjusted to meet the image resolution requirements of the multimodal model, and then input into the multimodal model to directly obtain the description information of the target object.

[0156] This concludes Figure 4 Description.

[0157] Please refer to Figure 5 , Figure 5 It is a schematic structural diagram of an electronic device proposed in an embodiment of the present application. At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a terminal interaction device at the logical level. Of course, in addition to software implementation methods, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0158] Please refer to Figure 6 , Figure 6 This is a structural diagram of a multi-modal small target detection device proposed in the embodiment of this application. Figure 6As shown, the apparatus may include a detection unit 601, a determination unit 602, and an inference unit 603. Specifically, the apparatus includes:

[0159] The detection unit 601 is configured to perform target detection on the image to be tested using the deployed target detection model to obtain location information of the target object in the image to be tested; the target object corresponds to the target category involved in the detection using the current multimodal model;

[0160] A determination unit 602 is configured to determine whether the target object satisfies the small target requirement based on the position information of the target object in the image to be measured and the image size information of the image to be measured, and if so, obtain a target object region based on the image input requirements of the multimodal model and the target object in the image to be measured; the target object region refers to a region that at least includes the target object;

[0161] The inference unit 603 is configured to input the image to be tested and the target object region into the multimodal model to obtain description information about the target object.

[0162] Optionally, the detection unit 601 is specifically configured to:

[0163] Using the deployed object detection model to perform object detection on the image to be tested, and obtain object position information; the object position information includes position information of at least one object in the image to be tested;

[0164] Based on the target category currently requiring multimodal model detection, determining the position information of the target object corresponding to the target category in the image to be tested from the obtained object position information;

[0165] And / or, the determining unit 602 is specifically configured to:

[0166] Determining the occupied size information of the target object in the image to be measured based on the position information of the target object in the image to be measured;

[0167] Determine whether the target object meets the small target requirement based on the occupied size information and the image size information of the image to be measured;

[0168] And / or, the image input requirement of the multimodal model includes an image aspect ratio requirement; the determining unit 602 is specifically configured to:

[0169] Selecting a reference area from the image to be tested; the reference area includes the target object and the background image around the target object;

[0170] Check whether the aspect ratio of the reference area matches the image aspect ratio requirement;

[0171] If the aspect ratio of the reference area does not match the image aspect ratio requirement, selecting a target object area including the reference area from the image to be measured; the aspect ratio of the target object area matches the image aspect ratio requirement;

[0172] And / or, the image input requirements of the multimodal model include image resolution requirements; the reasoning unit 603 is specifically used to:

[0173] Check whether the target object area and the resolution of the image to be tested match the image resolution requirements;

[0174] If the resolution of the target object region and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object region and / or the image to be measured so that the adjusted resolution of the target object region and the image to be measured match the image resolution requirement;

[0175] The adjusted target object region is used as a prompt image of the adjusted image to be tested, and the adjusted target object region and the adjusted image to be tested are detected based on the multimodal model to obtain description information about the target object;

[0176] And / or, the image input requirements of the multimodal model include image resolution requirements; the reasoning unit 603 is specifically used to:

[0177] Adjusting the resolution of the target object area and / or the image to be measured so that the adjusted target object area and the image to be measured meet the image stitching requirements;

[0178] Stitching the target object area that meets the image stitching requirements and the image to be tested to obtain a stitched image;

[0179] Check whether the resolution of the stitched image matches the image resolution requirement;

[0180] If the resolution of the target object region and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object region and / or the image to be measured so that the adjusted resolution of the target object region and the image to be measured match the image resolution requirement;

[0181] Inputting the spliced image into the multimodal model to detect the spliced image based on the multimodal model to obtain description information about the target object;

[0182] And / or, the image input requirement of the multimodal model includes an image resolution requirement; the reasoning unit 603 is further configured to:

[0183] If it is determined based on the position information of the target object in the image to be measured and the image size information of the image to be measured that the target object does not meet the small target requirement, then detecting whether the resolution of the image to be measured matches the image resolution requirement;

[0184] If not, the resolution of the image to be tested is adjusted to obtain the image to be input, and the resolution of the image to be input matches the image resolution requirement;

[0185] The input image is input into the multimodal model to obtain descriptive information about the target object that does not meet the small target requirements.

[0186] So far, completed Figure 6 Description of the multimodal small target detection device.

[0187] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed, the method disclosed in the above example of the present application can be implemented.

[0188] Exemplarily, the computer-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the computer-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0189] The above are merely preferred embodiments of the present application and are not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A multimodal small target detection method, characterized in that: The method includes: Performing target detection on the image to be tested using the deployed target detection model to obtain location information of the target object in the image to be tested; the target object corresponds to the target category involved in detection using the current multimodal model; Based on the position information of the target object in the image to be measured and the image size information of the image to be measured, determining whether the target object is an object that meets the small target requirement; if so, obtaining a target object area according to the image input requirements of the multimodal model and the target object in the image to be measured; the target object area refers to an area that at least includes the target object; The image to be measured and the target object region are input into the multimodal model to obtain description information about the target object.

2. The method according to claim 1, characterized in that The method of performing target detection on the image to be tested using the deployed target detection model to obtain the position information of the target object in the image to be tested includes: Performing target detection on the image to be tested using the deployed target detection model to obtain object position information; the object position information includes position information of at least one object in the image to be tested in the image to be tested; Based on the target category currently requiring multimodal model detection, position information of the target object corresponding to the target category in the image to be detected is determined from the obtained object position information.

3. The method according to claim 1, characterized in that Determining whether the target object meets the small target requirement based on position information of the target object in the image to be measured and image size information of the image to be measured includes: Determining the occupied size information of the target object in the image to be measured based on the position information of the target object in the image to be measured; It is determined whether the target object meets the small target requirement according to the occupied size information and the image size information of the image to be measured.

4. The method according to claim 1, wherein The image input requirement of the multimodal model includes an image aspect ratio requirement; obtaining a target object area according to the image input requirement of the multimodal model and the target object in the image to be measured includes: Selecting a reference area from the image to be measured; the reference area includes the target object and a background image around the target object; Detecting whether the aspect ratio of the reference area matches the image aspect ratio requirement; If the aspect ratio of the reference area does not match the image aspect ratio requirement, a target object area including the reference area is selected from the image to be measured; the aspect ratio of the target object area matches the image aspect ratio requirement.

5. The method according to claim 1, wherein The image input requirements of the multimodal model include image resolution requirements; inputting the image to be measured and the target object area into the multimodal model to obtain description information about the target object includes: Detecting whether the target object area and the resolution of the image to be measured match the image resolution requirement; If the resolution of the target object area and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object area and / or the image to be measured so that the adjusted resolution of the target object area and the image to be measured match the image resolution requirement; The adjusted target object region is used as a prompt image of the adjusted image to be tested, and the adjusted target object region and the adjusted image to be tested are detected based on the multimodal model to obtain description information about the target object.

6. The method according to claim 1, characterized in that The image input requirements of the multimodal model include image resolution requirements; inputting the image to be measured and the target object area into the multimodal model to obtain description information about the target object includes: Adjusting the resolution of the target object area and / or the image to be measured so that the adjusted target object area and the image to be measured meet the image stitching requirements; Stitching the target object area that meets the image stitching requirements and the image to be tested to obtain a stitched image; Detecting whether the resolution of the stitched image matches the image resolution requirement; If the resolution of the target object area and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object area and / or the image to be measured so that the adjusted resolution of the target object area and the image to be measured match the image resolution requirement; The spliced image is input into the multimodal model to detect the spliced image based on the multimodal model to obtain description information about the target object.

7. The method according to claim 1, characterized in that The image input requirements of the multimodal model include image resolution requirements; the method further includes: If it is determined based on the position information of the target object in the image to be measured and the image size information of the image to be measured that the target object does not meet the small target requirement, detecting whether the resolution of the image to be measured matches the image resolution requirement; If not, adjusting the resolution of the image to be tested to obtain an image to be input, wherein the resolution of the image to be input matches the image resolution requirement; The image to be input is input into the multimodal model to obtain description information about the target object that does not meet the small target requirement.

8. A multimodal small target detection device, characterized in that: The device includes: A detection unit, configured to perform target detection on the image to be tested using the deployed target detection model to obtain location information of the target object in the image to be tested; the target object corresponds to the target category involved in detection using the current multimodal model; a determining unit, configured to determine whether the target object satisfies the small target requirement based on position information of the target object in the image to be measured and image size information of the image to be measured, and if so, obtain a target object region according to the image input requirements of the multimodal model and the target object in the image to be measured; the target object region refers to a region that at least includes the target object; An inference unit is used to input the image to be measured and the target object area into the multimodal model to obtain description information about the target object.

9. The device according to claim 8, characterized in that The detection unit is specifically used for: Performing target detection on the image to be tested using the deployed target detection model to obtain object position information; the object position information includes position information of at least one object in the image to be tested in the image to be tested; Based on the target category currently requiring multimodal model detection, determining, from the obtained object position information, position information of a target object corresponding to the target category in the image to be tested; And / or, the determining unit is specifically configured to: Determining the occupied size information of the target object in the image to be measured based on the position information of the target object in the image to be measured; determining whether the target object meets the small target requirement according to the occupied size information and the image size information of the image to be measured; And / or, the image input requirement of the multimodal model includes an image aspect ratio requirement; and the determining unit is specifically configured to: Selecting a reference area from the image to be measured; the reference area includes the target object and a background image around the target object; Detecting whether the aspect ratio of the reference area matches the image aspect ratio requirement; If the aspect ratio of the reference area does not match the image aspect ratio requirement, selecting a target object area including the reference area from the image to be measured; the aspect ratio of the target object area matches the image aspect ratio requirement; And / or, the image input requirement of the multimodal model includes an image resolution requirement; and the inference unit is specifically configured to: Detecting whether the target object area and the resolution of the image to be measured match the image resolution requirement; If the resolution of the target object area and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object area and / or the image to be measured so that the adjusted resolution of the target object area and the image to be measured match the image resolution requirement; Using the adjusted target object region as a prompt image of the adjusted image to be tested, and detecting the adjusted target object region and the adjusted image to be tested based on the multimodal model to obtain description information about the target object; And / or, the image input requirement of the multimodal model includes an image resolution requirement; and the inference unit is specifically configured to: Adjusting the resolution of the target object area and / or the image to be measured so that the adjusted target object area and the image to be measured meet the image stitching requirements; Stitching the target object area that meets the image stitching requirements and the image to be tested to obtain a stitched image; Detecting whether the resolution of the stitched image matches the image resolution requirement; If the resolution of the target object area and / or the image to be measured does not match the image resolution requirement, adjusting the resolution of the target object area and / or the image to be measured so that the adjusted resolution of the target object area and the image to be measured match the image resolution requirement; Inputting the spliced image into the multimodal model to detect the spliced image based on the multimodal model to obtain description information about the target object; And / or, the image input requirement of the multimodal model includes an image resolution requirement; and the reasoning unit is further configured to: If it is determined based on the position information of the target object in the image to be measured and the image size information of the image to be measured that the target object does not meet the small target requirement, detecting whether the resolution of the image to be measured matches the image resolution requirement; If not, adjusting the resolution of the image to be tested to obtain an image to be input, wherein the resolution of the image to be input matches the image resolution requirement; The image to be input is input into the multimodal model to obtain description information about the target object that does not meet the small target requirement.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images