Target detection method, device, equipment and storage medium based on visual prompts

Abstract

The present application discloses a method, device, equipment and storage medium for target detection based on visual cues, the method comprising obtaining and selecting at least one visual cue image; obtaining visual cue information based on the visual cue image, wherein the visual cue information comprises cue information of at least one target to be detected in the visual cue image; and displaying target detection results that meet the target to be detected in the image to be detected based on the visual cue image and the visual cue information. The embodiment of the present application obtains visual cue information in a visual cue image, and then uses the visual cue image and the visual cue information as visual cues to detect targets that meet the target to be detected in the image to be detected according to the visual cues. In this way, any target can be detected without relying on predefined categories, thereby increasing the scope of application of the detection method, avoiding the manpower and time costs required for repeated training, and reducing the detection cost of target detection.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for target detection based on visual cues. Background Art

[0002] With the development of image processing technology, detection algorithms based on computer vision have become a hot research direction, and target detection algorithms have been widely used in industrial manufacturing and other fields. For example, in scenes with dense crowds or dense products such as intersections, farmlands, and factories, it is often necessary to perform target detection and target detection on target objects, so as to guide the implementation of relevant measures for public safety and production management.

[0003] Existing target detection methods are mainly implemented based on trained target detection models, that is, a deep learning model is trained using a closed data set to obtain a trained target detection model, and then the target detection model is used to process the image to obtain target detection. However, this method can only detect targets in the predefined categories in the target detection model, and cannot detect targets outside the predefined categories in the target detection model, which limits the applicable scenarios of the detection method and makes the detection method have adaptability problems. At the same time, if it is necessary to detect targets outside the predefined categories in the target detection model, it is necessary to re-collect training data and re-train the model, which requires re-spending time and manpower, increasing the detection cost of target detection.

[0004] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by the present application is to provide a target detection method, device, equipment and storage medium based on visual cues in view of the deficiencies in the prior art.

[0006] In order to solve the above technical problems, the first aspect of the embodiment of the present application provides a target detection method based on visual cues, wherein the method includes:

[0007] Select at least one visual cue image;

[0008] Acquiring visual prompt information based on the visual prompt image, wherein the visual prompt information includes prompt information of at least one target to be inspected in the visual prompt image;

[0009] The target detection result that matches the target to be detected in the image to be detected is displayed based on the visual prompt image and the visual prompt information.

[0010] The target detection method based on visual cues, wherein the prompt information includes the prompt position information of the target to be detected, or the prompt position information and the prompt category of the target to be detected, wherein the prompt category includes a positive example category or a negative example category, the positive example category indicates that the target to be detected is included in the target detection result, and the negative example category indicates that the target to be detected is not included in the target detection result.

[0011] The target detection method based on visual cues, wherein the target detection result includes at least one detection target in the image to be detected that meets the target to be detected, and the detection position information of each detection target, and when the prompt information includes a prompt category, the target detection result includes the detection position information of each detection target and / or the prompt category of each detection target.

[0012] The target detection method based on visual cues, wherein the target detection result includes one or more of the confidence level of the detected target, a display threshold, and a counting result of all the detected targets.

[0013] The target detection method based on visual cues, wherein, when the target detection result includes the confidence level and display threshold of each detected target, after displaying the target detection result corresponding to the image to be detected, the method further includes:

[0014] The display threshold is modified, and the target detection result is adjusted based on the modified display threshold.

[0015] In the target detection method based on visual cues, the display mode of the detection position information includes one or more of a detection box, a detection point and a detection segmentation.

[0016] The method for target detection based on visual cues, wherein, after displaying the target detection result corresponding to the image to be detected, the method further comprises:

[0017] Receiving an adjustment operation corresponding to the visual prompt information;

[0018] Adjusting the visual prompt information based on the adjustment operation;

[0019] The adjusted visual prompt information is used as visual prompt information, and the step of displaying the target detection result that matches the target to be detected in the image to be detected based on the visual prompt image and the visual prompt information is re-executed.

[0020] The target detection method based on visual cues, wherein the adjustment operation includes one or more of an adding operation, a deleting operation and a modifying operation, wherein the adding operation is used to add prompt information to the visual cue information, the deleting operation is used to delete the prompt information in the visual cue information; and the modifying operation is used to modify the prompt information in the visual cue information.

[0021] In the target detection method based on visual cues, the step of selecting at least one visual cue image specifically comprises:

[0022] Receiving an image selection operation input by a user, wherein the image selection operation includes a drag operation and / or a selection operation;

[0023] The image selected by the image selection operation is read to obtain at least one visual prompt image.

[0024] The method for target detection based on visual cues, wherein the step of obtaining visual cue information based on the visual cue image specifically comprises:

[0025] Receiving a prompt information selection operation input by a user, wherein the prompt information selection operation includes a marking operation performed on the visual prompt image and / or a prompt text input operation;

[0026] Based on the prompt information selection operation, a target to be inspected is selected in the visual prompt image, and prompt information of the selected target to be inspected is obtained.

[0027] The method for target detection based on visual cues, wherein the process of determining the target detection result specifically includes:

[0028] Extracting a first feature map of the visual cue image and a second feature map of the image to be inspected;

[0029] Determining a visual cue feature based on the first feature map and the visual cue information;

[0030] Based on the visual cue feature and the second feature map, an object detection result that matches the object to be detected in the image to be detected is determined.

[0031] In the method for object detection based on visual cues, the first feature map and the second feature map are both extracted by an image encoder, and the process of extracting the feature map by the image encoder specifically includes:

[0032] Inputting the target image into a backbone network in a preset image encoder, and extracting a visual feature map corresponding to the target image through the backbone network;

[0033] The visual feature map is input into a feature encoder in the image encoder, and a target feature map is output, wherein the feature encoder includes a plurality of cascaded Transformer encoders, and the Transformer encoder includes a multi-head self-attention layer and a feedforward neural network layer.

[0034] The method for object detection based on visual cues, wherein the step of determining visual cue features based on the first feature map and the visual cue information specifically comprises:

[0035] Inputting the visual cue information into a linear projection layer in a preset visual cue encoder, and obtaining a position code corresponding to the visual cue information through the linear projection layer;

[0036] The position code and the first feature map are input into a feature extraction module in the visual cue encoder, and the visual cue feature is determined by the feature extraction module, wherein the feature extraction module uses the position code as a query vector, a value vector and a key vector, and performs feature extraction on the first feature map through a self-attention mechanism to obtain the visual cue feature.

[0037] The method for detecting an object based on visual cues, wherein, based on the visual cues feature and the second feature map, determining the object detection result in the image to be detected that meets the object to be detected specifically comprises:

[0038] The visual cue feature and the second feature map are input into a preset target decoder, and a target detection result that matches the target to be detected in the image to be detected is determined by the target decoder, wherein the target decoder includes a Transformer decoder.

[0039] A second aspect of an embodiment of the present application provides an interactive target detection device based on visual cues, the interactive target detection device based on visual cues comprising:

[0040] A selection module, used for selecting at least one visual prompt image;

[0041] An acquisition module, configured to acquire visual prompt information based on the visual prompt image, wherein the visual prompt information includes prompt information of at least one target to be inspected in the visual prompt image;

[0042] A display module is used to display the target detection result that matches the target to be detected in the image to be detected based on the visual prompt image and the visual prompt information.

[0043] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the visual cue-based target detection methods described above.

[0044] A fourth aspect of the embodiments of the present application provides a terminal device, which includes: a processor and a memory;

[0045] The memory stores a computer-readable program executable by the processor;

[0046] When the processor executes the computer-readable program, the steps in any of the above-mentioned methods for detecting objects based on visual cues are implemented.

[0047] Beneficial effect: Compared with the prior art, the present application provides a target detection method, device, equipment and storage medium based on visual cues, the method comprising obtaining and selecting at least one visual cue image; obtaining visual cue information based on the visual cue image, wherein the visual cue information comprises cue information of at least one target to be detected in the visual cue image; and displaying the target detection result that meets the target to be detected in the image to be detected based on the visual cue image and the visual cue information. The embodiment of the present application obtains visual cue information from a visual cue image, and then uses the visual cue image and the visual cue information as visual cues to detect targets that meet the target to be detected in the image to be detected according to the visual cues. In this way, any target can be detected without relying on predefined categories, thereby improving the applicability of the detection method, avoiding the manpower and time costs required for repeated training, and reducing the detection cost of target detection. At the same time, after displaying the target detection result corresponding to the image to be detected, the present application receives an adjustment operation corresponding to the visual cue information, adjusts the visual cue information based on the adjustment operation, uses the adjusted visual cue information as the visual cue information, and re-executes the step of displaying the target detection result that meets the target to be detected in the image to be detected based on the visual cue image and the visual cue information. The target detection result can be adjusted according to visual feedback, thereby improving the accuracy and reliability of the detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1A flowchart of a method for detecting a target based on visual cues provided in an embodiment of the present application.

[0050] Figure 2 This is a principle flow chart of a specific implementation method of the target detection method based on visual cues in an embodiment of the present application.

[0051] Figure 3 This is a workflow comparison diagram of three specific working scenarios of the target detection method based on visual cues in an embodiment of the present application.

[0052] Figure 4 This is a schematic diagram of the structure of an interactive target detection device based on visual cues according to an embodiment of the present application.

[0053] Figure 5 This is a schematic diagram of the structure of the terminal device provided in this application. DETAILED DESCRIPTION

[0054] The present application provides a method, device, equipment and storage medium for object detection based on visual cues. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0057] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0058] After research, it is found that with the development of image processing technology, detection algorithms based on computer vision have become a hot research direction, and target detection algorithms have been widely used in industrial manufacturing and other fields. For example, for scenes with dense crowds or dense products such as intersections, farmlands, and factories, it is often necessary to perform target detection and target detection on the target objects, so as to guide the implementation of relevant measures for public safety and production management.

[0059] Existing target detection methods are mainly implemented based on trained target detection models, that is, a deep learning model is trained using a closed data set to obtain a trained target detection model, and then the target detection model is used to process the image to obtain target detection. However, this method can only detect targets in the predefined categories in the target detection model, and cannot detect targets outside the predefined categories in the target detection model, which limits the applicable scenarios of the detection method and makes the detection method have adaptability problems. At the same time, if it is necessary to detect targets outside the predefined categories in the target detection model, it is necessary to re-collect training data and re-train the model, which requires re-spending time and manpower, increasing the detection cost of target detection.

[0060] In order to solve the above problems, in an embodiment of the present application, at least one visual cue image is selected; visual cue information is obtained based on the visual cue image, wherein the visual cue information includes cue information of at least one target to be detected in the visual cue image; and target detection results that meet the target to be detected in the image to be detected are displayed based on the visual cue image and the visual cue information. The embodiment of the present application obtains visual cue information in the visual cue image, and then uses the visual cue image and the visual cue information as visual cues, and then detects the detection target that meets the target to be detected in the image to be detected based on the visual cue. In this way, any target can be detected without relying on predefined categories, thereby increasing the scope of application of the detection method, avoiding the manpower and time costs required for repeated training, and reducing the detection cost of target detection.

[0061] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0062] This embodiment provides a method for object detection based on visual cues, which can be applied to many different fields such as retail, transportation, and agriculture. Figure 1 As shown, the method includes:

[0063] S10: Select at least one visual prompt image.

[0064] Specifically, at least one visual cue image may include one visual cue image, or may include multiple visual cue images. For example, at least one visual cue image includes an image A, or at least one visual image includes an image A and an image B, etc. Of course, in practical applications, the number of visual cue images can be determined according to actual needs, for example, the number of images is 3, 5, 7, 8, etc. In addition, each visual cue image in at least one visual cue image is used to provide a cue feature for detecting a target, that is, each visual cue image includes at least one target object, and the target object can be used as a basis for detecting the image to be inspected. For example, the visual cue image is an image containing a seagull, or the visual cue image is an image containing a kitten and a puppy, etc.

[0065] In an implementation of the embodiment of the present application, the selecting at least one visual prompt image specifically includes:

[0066] Receiving an image selection operation input by a user, wherein the image selection operation includes a drag operation and / or a selection operation;

[0067] The image selected by the image selection operation is read to obtain at least one visual prompt image.

[0068] Specifically, the image selection operation is used to select a visual cue image, wherein the image selection operation can be a drag operation, an upload operation, or a selection operation. That is to say, the terminal device for running the target detection method based on visual cues provided in the embodiment of the present application can first have an interface for adding visual cue images, and then the user drags the visual cue information to the interface by dragging, so that the terminal device reads the visual cue image. Or, the terminal device has an existing image selection window, and the user forms an image selection operation by clicking on the image or inputting the image storage address, and then the terminal device can read the image selected by the image selection operation, and use the selected image as a visual cue image; or the terminal device has an existing image upload window, and the user clicks the upload operation, and then the terminal device can read the image uploaded by the image upload operation, and use the uploaded image as a visual cue image. Of course, it is worth noting that when there are multiple visual cue images, the visual cue image can be selected by one or more drag operations / selection operations / upload operations, or the visual cue image can be selected by combining drag operations, upload operations and selection operations. In addition, it is worth noting that the visual cue image selected through the image selection operation can be pre-provided by the terminal device running the visual cue-based target detection method, or it can be provided by the user himself, for example, transmitted from the user's own storage device to the terminal device, or downloaded to the terminal device via the network, or downloaded to the terminal device via the cloud, etc.

[0069] S20. Acquire visual prompt information based on the visual prompt image, wherein the visual prompt information includes prompt information of at least one target to be inspected in the visual prompt image.

[0070] Specifically, the visual prompt information is used as a detection basis for the image to be detected. That is to say, based on the prompt information of the target to be detected, the detection target that meets the target to be detected is detected in the image to be detected. In this way, the detection target can be directly specified through the visual prompt information. On the one hand, the description limitations of the text prompt information are avoided and the applicable scenarios of the detection method are improved. On the other hand, by directly specifying the target to be detected and obtaining the visual prompt information through the visual prompt image, the accuracy of the visual prompt information can be improved, and the user can easily and conveniently determine the visual prompt information.

[0071] The visual cue information includes the cue information of at least one target to be inspected in the visual cue image, that is, it may include the cue information of one target to be inspected, or it may include the cue information of multiple targets to be inspected. When the visual cue information includes the cue information of one target to be inspected, the cue information of the target to be inspected may be determined by one or more visual cue images; when the visual cue information includes the cue information of multiple targets to be inspected, the multiple targets to be inspected may be included in one visual cue image, or the multiple targets to be inspected may be included in multiple visual cue images. In addition, when multiple targets to be inspected may be included in multiple visual cue images, each target to be inspected may be associated with the visual cue image containing each target to be inspected, so as to determine the visual cue image corresponding to the cue information of each target to be inspected. In this way, when the target detection results are subsequently displayed based on the visual cue image and the visual cue information, the correspondence between each visual cue image and each cue information may be quickly known, thereby avoiding the problem of confusion between the cue information and the visual cue image, thereby improving the accuracy of target detection.

[0072] For example: assuming that the visual cue information includes cue information a corresponding to the target 1 to be inspected, and cue information b corresponding to the target 2 to be inspected, and the target 1 to be inspected is contained in the visual cue image A, and the target 2 to be inspected is contained in the visual cue image B, then the cue information a corresponds to the visual cue image A, and the cue information b corresponds to the visual cue image B. When subsequently determining the visual cue feature based on the first feature map corresponding to the visual cue image and the visual cue information, a visual cue feature is determined based on the first feature map of the visual cue image A and the cue information a, and a visual cue feature is determined based on the first feature map of the visual cue image B and the cue information b, and all the determined visual cue features are used as the visual cue features corresponding to the visual cue information.

[0073] In one implementation of the embodiment of the present application, the prompt information of each target to be inspected includes prompt position information, and the prompt position information is used to reflect the position information of the target to be inspected in the visual prompt image. Based on the prompt position information, the target to be inspected can be located in the corresponding visual prompt image. Wherein, the prompt position information can be the position information of the marking point of the target to be inspected, or it can be the position information of the marking box of the target to be inspected. In other words, the prompt position information can be formed by a marking point method, or it can be formed by a marking box method. Correspondingly, the prompt position information is represented by two-dimensional coordinates, or it can be represented by four-dimensional coordinates, wherein, when the prompt position information is a two-dimensional coordinate, the prompt position information is formed by a marking point method, and the two-dimensional coordinates are the image coordinates of the marking point in the visual prompt image, and when the prompt position information is a four-dimensional coordinate, the prompt position information is formed by a marking box method, and the four-dimensional coordinates can include the length of the marking box, the width of the marking box, and the two-dimensional coordinates of the upper left corner of the marking box, etc.

[0074] In one implementation of the embodiment of the present application, the prompt information of each target to be detected includes the prompt position information and the prompt category of the target to be detected, wherein the prompt position information has the same meaning as the prompt position information in the above implementation, and the prompt category is used to reflect whether the detection target that meets the target to be detected and corresponds to the prompt position information is included in the target detection result. Specifically, the prompt category includes a positive example category or a negative example category, and the positive example category indicates that the detection target that meets the target to be detected is included in the target detection result, and the negative example category indicates that the detection target that meets the target to be detected is not included in the target detection result. The embodiment of the present application adds a prompt category to the prompt position information, and then distinguishes the detection targets that meet the target to be detected based on the visual prompt information through the prompt category, so as to avoid the influence of detection targets with high similarity on the target detection results, thereby improving the accuracy of target detection.

[0075] Since the embodiment of the present application is to perform target detection, in order to avoid the situation where the detection result is not obtained based on the visual prompt information. In the embodiment of the present application, when the prompt information includes prompt position information and prompt category, there is at least one prompt information in the prompt information included in the visual prompt information, and the prompt category in the prompt information is a positive example category. In other words, when the visual prompt information only includes one prompt information, the prompt category included in the prompt information is a positive example category, and when the visual prompt information includes multiple prompt information, at least one prompt category included in the multiple prompt information is a positive example category. For example, the visual prompt information includes prompt information a and prompt information b, then the prompt category in prompt information a is a positive example category, or the prompt category in prompt information b is a positive example category, or the prompt category in prompt information a and the prompt category in prompt information b are both positive example categories.

[0076] In one embodiment, when detecting target A1 in the image to be detected, since the similarity between target A1 and target B1 in the image to be detected is higher than a preset similarity value (e.g., 80%), there may be a situation where target B1 is mistakenly detected as target A1 as the final detection target, resulting in inaccurate detection results.

[0077] Furthermore, in order to improve the accuracy of the detection results, the embodiment of the present application provides prompt information a for the target 1 to be detected as target A1, the prompt information a includes prompt position information a1 and prompt category a2, wherein prompt category a2 is a positive example category; there is prompt information b for the target 2 to be detected as target B1, the prompt information b includes prompt position information b1 and prompt category b2, wherein prompt category b2 is a negative example category; based on the prompt position information a1, detection is performed to obtain detection target results A1 and B1, and based on the prompt position information b1 and the prompt category b2, the detection target B1 can be removed from the obtained detection target results to obtain target A1, thereby improving the detection accuracy of target A.

[0078] In an embodiment of the present application, obtaining visual prompt information based on the visual prompt image specifically includes:

[0079] Receive prompt information selection operation input by the user;

[0080] Based on the prompt information selection operation, a target to be inspected is selected in the visual prompt image, and prompt information of the selected target to be inspected is obtained.

[0081] Specifically, the prompt information selection operation includes a marking operation and / or a prompt text input operation performed on the visual prompt image, wherein the marking operation is used to select the target to be inspected in the visual prompt image, and the prompt text input operation user inputs the description information of the target to be inspected. Further, the prompt information of the target to be inspected can be determined by directly extracting the coordinates of the annotations formed by the marking operation and extracting the prompt category. It can also be generated by the target detection model obtained through training according to the prompt text input operation to form operation information in the visual prompt image. In an embodiment of the present application, the marking operation can generate prompt position information by adopting a manual marking method (including directly outlining the marking point or marking box on the visual prompt image, or, by inputting the prompt information to form the marking point or marking box on the visual prompt image), and when the prompt information includes the prompt category, a prompt list is added to the prompt position information, so that the target detection can be converted into an open set of target detection tasks, so that the user can detect any target without relying on a specific category set, avoiding the problem of detection target category restriction caused by relying on a specific category set, and improving the scope of application of target detection. At the same time, it also saves the human and time resources required for training the detection model based on a specific category set, reducing the human and time resources required for target detection.

[0082] S30: Displaying the target detection result that matches the target to be detected in the image to be detected based on the visual prompt image and the visual prompt information.

[0083] Specifically, the image to be detected can be an image applied to various scenes. Accordingly, the target to be detected for detecting the image to be detected can be a person, a vehicle, an animal, a plant, or a commodity, etc., and the detection target in the image to be detected that meets the target to be detected can be a person, a vehicle, an animal, a plant, or a commodity, etc.

[0084] Further, the visual cue image can be one or more, and the image to be detected can be the same as one of the visual cue images, or it can be different from all visual cue images. In other words, when the visual cue image is one, the image to be detected can be the same as the visual cue image, or it can be different from the visual cue image, that is, the target detection method provided by the embodiment of the present application can use a target in the image to be detected as the target to be detected, to realize the target detection in the image, or, it can use a target in other images as the target to be detected, to realize the target detection across the image. Of course, when the visual image is a plurality of visual cue images, the image to be detected can be the same as a visual cue image in the plurality of visual cue images, or it can be different from a visual cue image in the plurality of visual cue images. When the user needs to improve the accuracy of target detection, the user can select a plurality of visual cue images to respectively perform the annotation operation of the prompt information of the target to be detected, and perform target detection with a plurality of visual cue information, so as to improve the accuracy and credibility of the detection result.

[0085] The embodiment of the present application can use an image that is the same as a visual cue image as the image to be detected, or can use an image that is different from each visual cue image as the image to be detected, so as to provide users with a variety of selection methods. When the user needs to improve the accuracy of target detection, the image that is the same as a visual cue image can be used as the image to be detected, so that the target detection is performed on the image to be detected by directly specifying the target to be detected in the image to be detected, so that accurate visual cues can be provided, thereby improving the accuracy of target detection. When the user wants to reduce the number of times the prompt information is annotated, the image that is different from each visual cue image can be used as the image to be detected, so that only the prompt information needs to be selected on the visual cue image, and target detection can be performed on other images to be detected that carry the target to be detected that corresponds to the prompt information, thereby reducing the human resources and time resources required for visual cue information annotation.

[0086] In one implementation of the embodiment of the present application, the target detection result is obtained by performing target detection on the image to be detected based on visual prompt information, wherein the target detection result in the image to be detected may include at least one detection target that meets the target to be detected, the detection position information of each detection target, and the detection position information of each detection target is used to reflect the position of each detection target in the image to be detected. Specifically, the target detection result may include a detection target result map, and each detection target and the detection position information of each detection target may be displayed in the detection target result map, wherein the detection target result map may be formed by adding the annotation information of the detection target to the image to be detected, or may be a detection target mask map corresponding to the image to be detected, etc. In other words, the display mode of the detection position information of each detection target in the target detection result may be one or more of a detection frame, a detection point, and a detection segmentation. The embodiment of the present application can improve the intuitiveness and user-friendliness of the target detection result by adopting a method of directly annotating the detection target in the image to be detected, which brings convenience to users.

[0087] In another implementation of the embodiment of the present application, when the prompt information includes a prompt category, the target detection result may include the detection position information of each detected target and / or the prompt category of each detected target. That is, the prompt information may include only the detection position information or only the prompt category, or may include both the detection position information and the prompt category.

[0088] In the embodiment of the present application, when the prompt information includes a prompt category, each detected detection target includes a prompt category. After the detection target is obtained according to the visual prompt information, the prompt category of the detection target in the image to be detected can be determined according to the prompt category included in each prompt information, and then each detection target is deleted according to the prompt category of each detection target, and finally each deleted detection target and its corresponding detection position information are used as the target detection result. In this way, the target detection result can only contain the detection targets of the target to be detected that meet the positive example category, thereby improving the intuitiveness of the target detection result and being easy to verify the detection result determined based on the prompt category. In addition, by carrying the prompt category in the target detection result, the embodiment of the present application can quickly determine which target detection results are displayed and which target detection results are not displayed, and can adjust the target detection result by switching the prompt category to be displayed.

[0089] In one implementation of the embodiment of the present application, in order to further improve the display effect of the target detection result, the target detection result may also include one or more of the confidence of the detection target, the display threshold and the counting results of all the detection targets. For example, the target detection result includes the confidence of the detection target, the display threshold and the counting results of all the detection targets, wherein the confidence of the detection target is the similarity between the detection target and the target to be detected. That is to say, the embodiment of the present application synchronously displays the detection position information, confidence, display threshold and the counting results of all the detection targets of each detection target in the target detection result, so that the user can intuitively know the conditions for determining the counting result, that is, the user can directly obtain which detection targets in the image to be detected participate in the counting, and the confidence of each detection target participating in the counting, thereby improving the intuitiveness of the target detection result and facilitating verification of the detection result determined based on the prompt category.

[0090] Furthermore, after displaying the target detection result, the user can adjust the display threshold according to their own needs, and then the target detection method provided in the embodiment of the present application will adjust the target detection result based on the adjusted display threshold. Based on this, when the target detection result includes the confidence and display threshold of each detected target, after displaying the target detection result corresponding to the image to be detected, the method further includes:

[0091] The display threshold is modified, and the target detection result is adjusted based on the modified display threshold.

[0092] Specifically, the confidence is used to reflect the similarity of the detection targets, and the display threshold is used to measure the confidence of the detection targets included in the target detection result, and the confidence of each detection target in the target detection result is greater than or equal to the display threshold. For example, the confidence value range of each detection target is 0-1. When the display threshold is 0.5, when determining the target detection result, the detection targets with a confidence greater than 0.5 are counted. When the display threshold is 0.6, when determining the target detection result, the detection targets with a confidence greater than 0.6 are counted.

[0093] In an embodiment of the present application, in order to better display the target detection result, a display interface corresponding to the target detection result can be set, and the target detection result can be displayed to the user through the display interface, wherein, on the display interface of the target detection result, there are a target detection result graph display area, a display threshold display area, and a counting result display area, wherein the target detection result graph display area is used to display the detection target, the detection position information and confidence of the detection target, the display threshold display area is used to display the display threshold, and the counting result display area is used to display the counting result. Of course, it is worth explaining that the display area on the display interface can be adjusted according to the result content included in the target detection result, wherein the result content includes displaying one or more of the detection target, the detection position information of the detection target, the confidence, the display threshold, and the counting result. For example, when the result content is to display the detection target and the detection position information of the detection target, only the target detection result graph display area can be set on the display interface.

[0094] Furthermore, after displaying the target detection result, the user can receive adjustment operations on the visual prompt information based on the target detection result, so that the user can make targeted adjustments to the visual prompt information according to the acquired target detection result, so as to adjust the wrong detection target, inaccurate detection target, and detection target that the user does not need in the target detection result, thereby improving the accuracy and credibility of the detection result. At the same time, the embodiment of the present application realizes an interactive detection process by displaying the target detection result to the user and receiving the user's adjustment operation, so that the user can control the calculation process and obtain a visualized detection result, which brings convenience to the user.

[0095] Based on this, in the embodiments of the present application, Figure 2 As shown, after displaying the target detection result corresponding to the image to be detected, the method further includes:

[0096] Receiving an adjustment operation corresponding to the visual prompt information;

[0097] Adjusting the visual prompt information based on the adjustment operation;

[0098] The adjusted visual prompt information is used as visual prompt information, and the step of displaying the target detection result that matches the target to be detected in the image to be detected based on the visual prompt image and the visual prompt information is re-executed.

[0099] Specifically, the adjustment operation is used to adjust the visual prompt information, wherein the operation includes one or more of an adding operation, a deleting operation and a modifying operation, wherein the adding operation is used to add prompt information to the visual prompt information, the deleting operation is used to delete the prompt information in the visual prompt information, and the modifying operation is used to modify the prompt information in the visual prompt information. Specifically, when the prompt information does not include a prompt category, the modifying operation is used to modify the prompt position information in the prompt information; when the prompt information includes a prompt category, the adding operation is used to add prompt information including a positive example category to the visual prompt information, and can also add prompt information including a negative example category to the visual prompt information, and can also add prompt information including a positive example category and prompt information including a negative example category to the visual prompt information at the same time; the modifying operation can be used to modify the prompt position information in the prompt information, and can also be used to modify the prompt category in the prompt information, or, can be used to modify the prompt position information and the prompt category in the prompt information at the same time.

[0100] For example: assuming that the visual cue information includes cue information a and cue information b, cue information a includes cue position information a1 and cue category a2 of the positive example category, cue information b includes cue position information b1 and cue category b2 of the negative example category, and the adjustment operation includes an adding operation, a deleting operation, and a modifying operation. The adding operation is used to add cue information c including cue position information c1 and cue category c2 of the positive example category and cue information d including cue position information d1 and cue category d2 of the positive example category, the deleting operation is used to delete cue information b, and the modifying operation is used to modify the cue position information a1 to cue position information a11. Then, the adjusted visual cue information includes cue information a including cue position information a11 and cue category a2 of the positive example category, c including cue position information c1 and cue category c2 of the positive example category, and cue information d including cue position information d1 and cue category d2 of the positive example category.

[0101] The embodiment of the present application improves the adjustment operation of visual prompt information for users. When part of the detected target in the image to be detected is not detected, prompt information carrying positive example categories can be added to the visual prompt information, or, when part of the target in the image to be detected is detected incorrectly, prompt information carrying negative example categories can be added to the visual prompt information; or, when part of the target in the image to be detected is detected incorrectly, and the visual prompt information carries prompt information corresponding to the part of the target, then the prompt information corresponding to the part of the target can be deleted from the visual prompt information. This allows users to freely adjust the target detection results, which brings convenience to users.

[0102] In addition, it is worth noting that the step of re-performing the step of displaying the target detection result that matches the target to be detected in the image to be detected based on the visual prompt image and the visual prompt information can also be adjusted according to the target detection result obtained this time, so as to determine the final target detection result through multiple rounds of interaction. The specific number of multiple rounds of interaction is not limited here and can be determined according to the actual usage of the user.

[0103] In an embodiment of the present application, the detection process of the target detection result can be performed by a terminal device running the target detection method provided in an embodiment of the present application, or by a background server or cloud device connected to the terminal device, wherein when executed by a background server or cloud device, the acquired visual prompt information, visual prompt image, and image to be detected can be sent to the background server or cloud device, the target detection result determined by the background server or cloud device is received, and the received target detection result is displayed. Here, the subject of determining the target detection result is not limited, and only the specific determination process of the target detection result is given.

[0104] Based on this, the process of determining the target detection result specifically includes:

[0105] S31, extracting a first feature map of the visual cue image and a second feature map of the image to be inspected;

[0106] S32, determining a visual prompt feature based on the first feature map and the visual prompt information;

[0107] S33: Determine, based on the visual cue feature and the second feature map, an object detection result that matches the object to be detected in the image to be detected.

[0108] Specifically, in step S31, the first feature map is used to reflect the image information carried by the visual cue image, and is used as the visual feature information of the visual cue image, and the second feature map is used to reflect the image information carried by the image to be detected, and is used as the visual feature information of the image to be detected. Among them, the first feature map and the second feature map can be extracted by a trained feature extraction model, for example, a deep learning model, a convolutional network model or a VGG model. In addition, it is worth noting that when the visual cue image is the same as the image to be detected, only one feature extraction is required to obtain the first feature map and the second feature map.

[0109] In the embodiments of the present application, Figure 2As shown, the first feature map and the second feature map are both extracted by a pre-trained image encoder, and rich visual features are extracted from the image to be detected and the visual cue image by the image encoder. Among them, the input item of the image encoder is the input image, and the output item is the feature map. That is to say, when it is necessary to determine the first feature map, the visual cue image is input into the image encoder, and when it is necessary to determine the second feature map, the image to be detected is input into the image encoder. Based on this, in order to facilitate the description of the process of feature extraction through the image encoder, the target image is recorded as the input image of the image encoder, and the target feature map is recorded as the output item of the image encoder.

[0110] Accordingly, the process of extracting feature maps by the image encoder may include:

[0111] Inputting the target image into a backbone network in an image encoder, and extracting a visual feature map corresponding to the target image through the backbone network;

[0112] The visual feature map is input into a feature encoder in an image encoder, and a target feature map is output through the feature encoder.

[0113] Specifically, the target image is an input item of the image encoder, wherein the target image can be an image to be detected or a visual cue image. The image encoder includes a backbone network and a feature encoder, wherein the backbone network is connected to the feature encoder, the backbone network is used to perform feature extraction on the target image to extract a visual feature map, and the feature encoder is used to capture the long-distance dependency between the features included in the visual feature map to extract a target feature map of the target image.

[0114] Further, the backbone network may adopt VGG and SwinTransformer, etc. In an embodiment of the present application, the backbone network adopts the SwinTransformer structure, and the multi-level visual image features of the target image with an image scale of HxWxC (H represents the height of the target image, W represents the width of the target image, and C represents the number of channels of the target image) are extracted by SwinTransformer, wherein the image dimensions in the visual feature maps of the multi-level visual image features are different, for example, the multi-level visual image features include 4 layers of visual image feature maps, the number of channels of each visual image feature map in the 4 layers of visual image feature maps is the same, which is equal to the number of channels of the target image, the image width and image height of each visual image feature map in the 4 layers of visual image feature maps are different, and the image length and image height of the 4 layers of visual image feature maps are respectively: (H / 2)*(W / 2), (H / 4)*(W / 4), (H / 8)*(W / 8) and (H / 16)*(W / 16).

[0115] Furthermore, the feature encoder includes several cascaded Transformer encoders, and the Transformer encoder includes a multi-head self-attention layer and a feedforward neural network layer. For example, the feature encoder includes a 6-layer Transformer encoder. Among them, the multi-head self-attention layer is used to capture the long-distance dependencies between features, and the feedforward neural network is used to perform feature transformation. By adopting a feature encoder including a Transformer encoder, the embodiment of the present application can enable the feature encoder to effectively process complex image content and extract deep and rich feature information.

[0116] In step S32, the visual cue feature is used as a basis for selecting the detection target, that is, the visual cue feature is used to reflect the visual features of the target to be detected, and in the subsequent target detection, the visual cue feature can be used as a basis for selecting the detection target in the image to be detected. Wherein, the visual cue information includes the prompt information, and the visual cue feature can be directly selected based on the prompt information in the first feature map, for example, the image area corresponding to the prompt information in the first feature map is directly used as the visual cue feature, or, based on the prompt information, the image area is determined in the first feature map in a preset manner (such as, with the prompt information as the center of the area, and the preset length as the length of the area, etc.), and the image area is used as the visual cue feature, or, the visual cue information can be used as guidance information, and the first feature map is encoded by a preset visual cue encoder to obtain the visual cue feature, etc.

[0117] In the embodiment of the present application, determining the visual prompt feature based on the first feature map and the visual prompt information specifically includes:

[0118] Inputting the visual cue information into a linear projection layer in a preset visual cue encoder, and obtaining a position code corresponding to the visual cue information through the linear projection layer;

[0119] The position code and the first feature map are input into a feature extraction module in the visual cue encoder, and the visual cue feature is determined by the feature extraction module, wherein the feature extraction module uses the position code as a query vector, a value vector and a key vector, and performs feature extraction on the first feature map through a self-attention mechanism to obtain the visual cue feature.

[0120] Specifically, the visual cue encoder is used to process the visual cue information to obtain visual cue features determined based on the visual cue information, wherein the linear projection layer is used to extract the position code corresponding to the visual cue information, and the position code is used to reflect the position and spatial information of the target to be inspected in the visual cue image. The feature extraction module is used to perform feature extraction on the first feature map based on the position code to obtain the visual features corresponding to the visual cue information.

[0121] Furthermore, since the visual cue information can be two-dimensional cue information in the form of annotated points or four-dimensional cue information in the form of annotated boxes, when the position code corresponding to the visual cue information is obtained through the linear projection layer, the corresponding linear projection layer can be determined according to the visual cue information, and then the visual cue information is projected into a position code through the linear projection layer corresponding to the visual cue information. In other words, the visual cue encoder can include several parallel linear projection layers, and the several parallel linear projection layers are all connected to the feature extraction module, and each of the several parallel linear projection layers corresponds to a formation method of the visual cue information, so that visual cue features can be extracted for visual cue information formed in different formation methods, so that users can use various different methods to annotate visual cue information, which brings convenience to users.

[0122] In the embodiment of the present application, the visual cue information is formed in the form of annotation points and annotation boxes. Correspondingly, the visual cue encoder includes two parallel linear projection layers, namely, the annotation box linear projection layer and the annotation box linear projection layer. bbox And the standard point linear projection layer Linear point , where Linear bbox and Linear point The projection process can be expressed as:

[0123] E bbox =Linear bbox (P)

[0124] E point =Linear point (P)

[0125] Among them, P represents visual prompt information, E bbox and E point Both represent positional encoding.

[0126] In the embodiment of the present application, the operation performed by the feature extraction module can be expressed as:

[0127] X bbox =Extract(F reference , E bbox )

[0128] X point =Extract(F reference , E point )

[0129] Among them, X bbox and X pointBoth represent visual cue features, Extract represents feature extraction operations, and F reference Represents the first feature map.

[0130] Furthermore, the feature extraction operation can be constructed using a multi-layer Transformer encoder. By using the position encoding as a query vector (Q), a value vector (V) and a key vector (K), a position-sensitive (i.e., using the position encoding as a query vector, a value vector and a key vector) feature extraction operation is performed on the first feature map through a multi-head self-attention layer in the Transformer encoder to capture image features related to the visual cue information to obtain visual cue features. The embodiment of the present application uses a multi-layer Transformer encoder to form a feature extraction module, and can use a multi-head self-attention layer to well capture image features related to the visual cue information, thereby determining the accuracy of the visual cue features, thereby improving the accuracy of target detection.

[0131] The visual cue encoder in the embodiment of the present application first encodes the visual cue information into a position code, and then uses the position code to perform feature extraction on the first feature map. In this way, by combining the position code with the first feature map, the image features of the target to be inspected located by the visual cue information can be located to obtain the visual cue features, so that the visual cue features include the visual information for the target to be inspected (such as color, texture, shape, etc.) and the contextual information of the target to be inspected in the visual cue image, so that target detection can be performed based on the visual cue information, thereby flexibly adapting to various detection tasks according to the specific needs of users, rather than being limited to predefined target categories.

[0132] In step S33, based on the second feature map and the visual cue feature, the visual cue feature can be directly used as a basis to select a feature area in the second feature map whose similarity with the visual cue feature reaches a preset similarity threshold, and then the target included in the image area corresponding to the feature area is used as the detection target, or the target detection result is determined by a pre-trained target decoder, that is, the second feature map and the visual cue feature are input into the target decoder, and the target detection result is output through the target decoder, etc.

[0133] In the embodiment of the present application, determining the target detection result that meets the target in the image to be inspected based on the visual cue feature and the second feature map specifically includes:

[0134] The visual cue feature and the second feature map are input into a preset target decoder, and a target detection result that matches the target to be detected in the image to be detected is determined by the target decoder, wherein the target decoder includes a Transformer decoder.

[0135] Specifically, the target decoder is used to mark the detection target in the image to be detected using the visual cue features and the second feature map to obtain the target detection result. Among them, the target decoder receives the visual cue information and the second feature map, and then identifies the detection target by decoding the visual cue features and the second feature map, and outputs the detection position information of each detected target. In addition, in practical applications, the confidence of each detected target can also be output. Correspondingly, the processing process of the target decoder can be expressed as:

[0136] B, S = Decoder (F target , X bbox ) / Decoder(F target , X point )

[0137] Among them, B represents the detection position information, S represents the confidence, Decoder represents the target decoder, and F target represents the second feature map, X bbox and X point Both represent visual cue features.

[0138] In an embodiment of the present application, the target decoder can be based on the DETR structure, that is, the target decoder includes a Transformer encoder, through the attention mechanism and the Transformer architecture, to achieve efficient target detection and positioning. Specifically, the target decoder includes multiple self-attention layers and a feedforward neural network, which are used for multiple self-attention layers to capture the correlation and feature space information of the visual cue features and the second feature map to generate accurate detection position information and confidence, which can improve the accuracy of the detection results. In addition, the embodiment of the present application only needs to perform a forward propagation through the decoder once, which improves the computational efficiency, thereby increasing the target detection speed.

[0139] The above completes the description of the process of determining the target detection result. Based on the above description, it can be known that in practical applications, the above-mentioned visual encoder, image encoder and target decoder can be made into a target detection model, wherein the image encoder is respectively connected to the visual encoder and the target decoder, the visual encoder is connected to the target decoder, the image encoder is used to extract the first feature map of the visual cue image and the second feature map of the image to be detected, the visual encoder determines the visual cue feature based on the first feature map and the visual cue information, and the target decoder is used to determine the target detection result that meets the target to be detected in the image to be detected based on the visual cue feature and the second feature map. Then, when in use, the target detection model is directly deployed on the execution body of the target detection result determination process, and the target detection model outputs the target detection result.

[0140] Several specific working scenarios are given below to further illustrate the target detection method based on visual cues provided in the embodiments of the present application.

[0141] Working scenario 1: Positive example prompt mode, that is, the image to be detected is used as the visual prompt image, and the visual prompt information does not include the prompt category. Working scenario 1 is suitable for scenes where the detection target is obvious and easy to distinguish, such as Figure 3 The scene shown. Specifically, in working scene 1, click once or draw a label box in the image to be detected (recorded as image A) to specify the target to be detected, and obtain the prompt information corresponding to the target to be detected to obtain visual prompt information, and then determine the target detection result corresponding to the image to be detected based on the visual prompt information and the visual prompt image. However, in the case of dense or small targets, one round of prompts may not be sufficient to detect all detection targets, making the target detection result inaccurate. Therefore, the user can add additional visual prompt information in the missing area according to the target detection result to obtain a more accurate target detection result.

[0142] Working scenario 2: Positive + negative prompt mode, that is, the image to be detected is used as the visual prompt image, and the prompt information includes the prompt category. Working scenario 2 is suitable for complex scenes where the detection target is similar to the surrounding environment or other targets in appearance, such as Figure 3 The scene shown. Specifically, in working scene 2, click once or draw a box in the image to be detected (recorded as image A) to specify the target to be detected, and obtain the prompt information corresponding to the target to be detected to obtain the visual prompt information, and then determine the target detection result corresponding to the image to be detected based on the visual prompt information and the visual prompt image. However, in complex scenes where the detection target is similar in appearance to the surrounding environment or other targets, other targets may be mistaken for the detection target. For example, the target to be detected corresponding to the visual prompt information is a green apple, and there is an orange citrus in the target detection result. Therefore, the user can add prompt information carrying a negative example category according to the target detection result to remove the wrong target in the target detection result.

[0143] Working scenario 3: Cross-image prompt mode, that is, the visual prompt image is different from the image to be detected. Working scenario 3 is suitable for automatic annotation and large-scale image processing scenarios, such as Figure 3 The scene shown. Specifically, in working scene 3, click once or draw a box in the visual prompt image (recorded as image B) to specify the target to be inspected, and obtain the prompt information corresponding to the target to be inspected to obtain the visual prompt information, and then determine the target detection result corresponding to the image to be detected (recorded as image A) based on the visual prompt information and the visual prompt image. Of course, it is worth noting that in working scene 3, after obtaining the target detection result, the visual prompt information can also be corrected.

[0144] In summary, the present embodiment provides a method, device, equipment and storage medium for target detection based on visual cues, the method comprising obtaining and selecting at least one visual cue image; obtaining visual cue information based on the visual cue image, wherein the visual cue information includes cue information of at least one target to be detected in the visual cue image; and displaying the target detection result that meets the target to be detected in the image to be detected based on the visual cue image and the visual cue information. The present embodiment of the application obtains visual cue information in the visual cue image, and then uses the visual cue image and the visual cue information as visual cues to detect the detection target that meets the target to be detected in the image to be detected according to the visual cues. In this way, any target can be detected without relying on predefined categories, thereby improving the scope of application of the detection method, avoiding the manpower and time costs required for repeated training, and reducing the detection cost of target detection. At the same time, the present embodiment of the application determines the visual cue information by specifying the cue information in the visual cue image. On the one hand, the visual cue information can be determined simply and quickly, and on the other hand, the situation where the detection target is difficult to describe can be avoided, thereby ensuring the accuracy of the visual cue information. In addition, after obtaining the target detection results, the embodiment of the present application displays the target detection results to the user and marks the detection target in the form of a labeling box. On the one hand, this allows the user to obtain an intuitive visual experience and can easily verify and understand the detection results determined based on the detection target, thereby improving the usability and acceptability of the detection results. On the other hand, the user can also correct the visual prompt information according to the target detection results. This allows the user to intervene in and correct the detection results, thereby not only improving the accuracy of the detection results, but also making the detection results more meet user needs.

[0145] Based on the above-mentioned target detection method based on visual cues, the embodiment of the present application provides an interactive target detection device based on visual cues, such as Figure 4 As shown, the interactive target detection device based on visual cues includes:

[0146] A selection module 100, for selecting at least one visual prompt image;

[0147] An acquisition module 200, configured to acquire visual prompt information based on the visual prompt image, wherein the visual prompt information includes prompt information of at least one target to be inspected in the visual prompt image;

[0148] The display module 300 is used to display the target detection result that matches the target to be detected in the image to be detected based on the visual prompt image and the visual prompt information.

[0149] Based on the above-mentioned target detection method based on visual cues, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the target detection method based on visual cues as described in the above-mentioned embodiment.

[0150] Based on the above-mentioned target detection method based on visual cues, the present application also provides a terminal device, such as Figure 5 As shown, it includes at least one processor (processor) 20; display screen 21; and memory (memory) 22, and may also include a communications interface (Communications Interface) 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0151] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0152] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0153] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0154] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for object detection based on visual cues, characterized in that: The method comprises: Select at least one visual cue image; Acquiring visual prompt information based on the visual prompt image, wherein the visual prompt information includes prompt information of at least one target to be inspected in the visual prompt image; Based on the visual cue image and the visual cue information, a target detection result that matches the target to be detected in the image to be detected is displayed, wherein the process of determining the target detection result specifically includes: Extracting a first feature map of the visual cue image and a second feature map of the image to be detected; Determining a visual cue feature based on the first feature map and the visual cue information; Based on the visual cue feature and the second feature map, an object detection result that matches the object to be detected in the image to be detected is determined.

2. The method for object detection based on visual cues according to claim 1, characterized in that: The prompt information includes prompt position information of the target to be detected, or prompt position information and prompt category of the target to be detected, wherein the prompt category includes a positive example category or a negative example category, the positive example category indicates that the target to be detected is included in the target detection result, and the negative example category indicates that the target to be detected is not included in the target detection result.

3. The method for object detection based on visual cues according to claim 1 or 2, characterized in that: The target detection result includes at least one detection target in the image to be detected that meets the target to be detected, and detection position information of each detection target, and when the prompt information includes a prompt category, the target detection result includes the detection position information of each detection target and / or the prompt category of each detection target.

4. The method for target detection based on visual cues according to claim 3, characterized in that: The target detection result includes one or more of the confidence level of the detected target, a display threshold, and a counting result of all the detected targets.

5. The method for target detection based on visual cues according to claim 4, characterized in that: When the target detection result includes the confidence level and display threshold of each detected target, after displaying the target detection result corresponding to the image to be detected, the method further includes: The display threshold is modified, and the target detection result is adjusted based on the modified display threshold.

6. The method for object detection based on visual cues according to claim 3, characterized in that: The display mode of the detection position information includes one or more of a detection frame, a detection point and a detection segmentation.

7. The method for object detection based on visual cues according to claim 1, characterized in that: After displaying the target detection result corresponding to the image to be detected, the method further includes: Receiving an adjustment operation corresponding to the visual prompt information; Adjusting the visual prompt information based on the adjustment operation; The adjusted visual prompt information is used as visual prompt information, and the step of displaying the target detection result that matches the target to be detected in the image to be detected based on the visual prompt image and the visual prompt information is re-executed.

8. The method for object detection based on visual cues according to claim 7, characterized in that: The adjustment operation includes one or more of an adding operation, a deleting operation and a modifying operation, wherein the adding operation is used to add prompt information to the visual prompt information, the deleting operation is used to delete the prompt information in the visual prompt information; and the modifying operation is used to modify the prompt information in the visual prompt information.

9. The method for object detection based on visual cues according to claim 1, characterized in that: The selecting at least one visual prompt image specifically includes: Receiving an image selection operation input by a user, wherein the image selection operation includes a drag operation and / or a selection operation; The image selected by the image selection operation is read to obtain at least one visual prompt image.

10. The method for object detection based on visual cues according to claim 1, characterized in that: The acquiring of visual prompt information based on the visual prompt image specifically includes: Receiving a prompt information selection operation input by a user, wherein the prompt information selection operation includes a marking operation performed on the visual prompt image and / or a prompt text input operation; Based on the prompt information selection operation, a target to be inspected is selected in the visual prompt image, and prompt information of the selected target to be inspected is obtained.

11. The method for object detection based on visual cues according to claim 1, characterized in that: The first feature map and the second feature map are both extracted by an image encoder, and the process of extracting the feature map by the image encoder specifically includes: Inputting the target image into a backbone network in a preset image encoder, and extracting a visual feature map corresponding to the target image through the backbone network; The visual feature map is input into a feature encoder in the image encoder, and a target feature map is output, wherein the feature encoder includes a plurality of cascaded Transformer encoders, and the Transformer encoder includes a multi-head self-attention layer and a feedforward neural network layer.

12. The method for object detection based on visual cues according to claim 1, characterized in that: The determining of the visual prompt feature based on the first feature map and the visual prompt information specifically includes: Inputting the visual cue information into a linear projection layer in a preset visual cue encoder, and obtaining a position code corresponding to the visual cue information through the linear projection layer; The position code and the first feature map are input into a feature extraction module in the visual cue encoder, and the visual cue feature is determined by the feature extraction module, wherein the feature extraction module uses the position code as a query vector, a value vector and a key vector, and performs feature extraction on the first feature map through a self-attention mechanism to obtain the visual cue feature.

13. The method for object detection based on visual cues according to claim 1, characterized in that: Determining, based on the visual cue feature and the second feature map, a target detection result that matches the target to be detected in the image to be detected specifically includes: The visual cue feature and the second feature map are input into a preset target decoder, and a target detection result that matches the target to be detected in the image to be detected is determined by the target decoder, wherein the target decoder includes a Transformer decoder.

14. An interactive target detection device based on visual cues, characterized in that: The interactive target detection device based on visual cues comprises: A selection module, used for selecting at least one visual prompt image; An acquisition module, configured to acquire visual prompt information based on the visual prompt image, wherein the visual prompt information includes prompt information of at least one target to be inspected in the visual prompt image; A display module is used to display the target detection result that meets the target to be detected in the image to be detected based on the visual prompt image and the visual prompt information, wherein the determination process of the target detection result specifically includes: Extracting a first feature map of the visual cue image and a second feature map of the image to be detected; Determining a visual cue feature based on the first feature map and the visual cue information; Based on the visual cue feature and the second feature map, an object detection result that matches the object to be detected in the image to be detected is determined.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the visual cue-based target detection method as described in any one of claims 1-13.

16. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the target detection method based on visual cues as described in any one of claims 1-13 are implemented.

Images