Target detection method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a target detection method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps of performing closed set target detection processing on a target image through a closed set target detection model to obtain a first target detection result, and performing open set target detection processing on the target image through an open set target detection model to obtain a second target detection result, the detection accuracy of the first target detection result is higher than that of the second target detection result, and performing fusion processing on the first target detection result and the second target detection result to obtain a target detection result of the target image. Through the method and the device, the accuracy of target detection can be improved under the condition of ensuring open set detection.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence technology, and particularly to a target detection method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Artificial intelligence (AI) is a comprehensive technology of computer science, which makes machines have the functions of perception, reasoning and decision-making by studying the design principles and implementation methods of various intelligent machines. Artificial intelligence technology is a comprehensive discipline, which involves a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] In the application scenario of target recognition of images, in order to make the target detection model recognize as many objects in the image as possible, a large amount of training data is often used to train the target detection model, so that the target detection model can "recognize" as many categories of images as possible. Although this method can recognize more categories of images, it cannot guarantee that all target objects in the image can be recognized, and a large amount of training samples are required for training. SUMMARY

[0004] The embodiments of the present application provide a target detection method, device, electronic equipment, computer readable storage medium and computer program product, which can improve the accuracy of target detection while ensuring open set detection.

[0005] The technical solution of the embodiments of the present application is as follows:

[0006] The embodiments of the present application provide a target detection method, which comprises:

[0007] performing closed set target detection processing on the target image through a closed set target detection model to obtain a first target detection result;

[0008] performing open set target detection processing on the target image through an open set target detection model to obtain a second target detection result;

[0009] The detection accuracy of the first target detection result is higher than that of the second target detection result.

[0010] fuse the first target detection result and the second target detection result to obtain a target detection result of the target image.

[0011] The embodiments of the present application provide a target detection device, which comprises:

[0012] a closed set detection module configured to perform closed set object detection processing on the target image by using a closed set object detection model to obtain a first object detection result;

[0013] an open set detection module configured to perform open set object detection processing on the target image by using an open set object detection model to obtain a second object detection result, wherein the detection accuracy of the first object detection result is higher than that of the second object detection result;

[0014] a fusion processing module configured to perform fusion processing on the first object detection result and the second object detection result to obtain an object detection result of the target image.

[0015] In the foregoing solution, the fusion processing module is further configured to perform splicing processing on the first object detection result and the second object detection result to obtain an object detection splicing result, and determine the object detection result of the target image based on the object detection splicing result.

[0016] In the foregoing solution, the fusion processing module is further configured to perform splicing processing on the first image block corresponding to the first detection box, the second image block corresponding to the second detection box, and the third image block corresponding to the third detection box to obtain a detection box splicing result, and perform splicing processing on the description parameters of the first detection box and the description parameters of the third detection box to obtain a parameter splicing result.

[0017] In the foregoing solution, the fusion processing module is further configured to perform similarity matching processing on the second detection box to obtain the description parameters of the second detection box, and perform splicing processing on the description parameters of the first detection box, the description parameters of the second detection box, and the description parameters of the third detection box to obtain a parameter splicing result.

[0018] In the foregoing solution, the fusion processing module is further configured to obtain a class feature of a candidate detection class, perform image feature extraction processing on the second image block corresponding to the second detection box to obtain an image feature of the second image block corresponding to the second detection box, determine a feature similarity between the image feature and the class feature, and if the feature similarity is higher than a similarity threshold, take the candidate detection class corresponding to the class feature as an image class of the second detection box, and take the feature similarity as a detection box confidence of the second detection box.

[0019] In the scheme, the fusion processing module is further configured to: acquire a keyword of the candidate detection category; generate a text description of the keyword based on the keyword of the candidate detection category; perform text feature extraction processing on the text description of the keyword to obtain a text feature of the text description of the keyword; and determine a category feature of the candidate detection category based on the text feature of the text description of the keyword.

[0020] In the scheme, the fusion processing module is further configured to: expand the keyword to obtain a synonym keyword of the keyword; and perform text generation processing on the keyword and the synonym keyword to obtain a first text description including the keyword and a second text description including the synonym keyword.

[0021] In the scheme, the fusion processing module is further configured to: perform word segmentation processing on the text description to obtain a vocabulary unit included in the text description; perform feature embedding processing on the vocabulary unit to obtain a unit embedding feature of the vocabulary unit; and perform splicing processing on unit embedding features of a plurality of the vocabulary units to obtain the text feature of the keyword.

[0022] In the scheme, the fusion processing module is further configured to: if the keyword has a plurality of the text descriptions, perform fusion processing on text features of the plurality of the text descriptions to obtain a keyword feature of the keyword; if the keyword has one of the text descriptions, determine the text feature of the text description as the keyword feature of the keyword; if the keyword has a synonym keyword, perform fusion processing on the keyword feature of the keyword and a keyword feature of the synonym keyword to obtain the category feature of the candidate detection category; and if the keyword does not have a synonym keyword, determine the keyword feature of the keyword as the category feature of the candidate detection category.

[0023] In the scheme, the fusion processing module is further configured to: perform target image block-based image segmentation processing on the target image to obtain at least one mask belonging to a target image block and a mask confidence of each of the masks, wherein the target image block is any image block from the detection frame splicing result; generate a bounding box corresponding to a first mask in the target image as a target detection frame, wherein the first mask is any one of the at least one mask; determine a target confidence of the target detection frame based on a detection frame confidence included in the parameter splicing result and a mask confidence of the first mask; and generate the target detection result based on the target detection frame and the target confidence.

[0024] In the foregoing scheme, the fusion processing module is further configured to perform standardization processing on the detection box confidence and the mask confidence of the first mask respectively to obtain a standardized detection box confidence and a standardized mask confidence; and multiply the standardized detection box confidence and the standardized mask confidence having a corresponding relationship to obtain the target confidence, where the corresponding relationship indicates that the first mask corresponding to the standardized mask confidence is in the detection box corresponding to the standardized detection box confidence.

[0025] An electronic device is provided in an embodiment of the present application, and the electronic device includes:

[0026] A memory is configured to store computer-executable instructions.

[0027] A processor is configured to execute the computer-executable instructions stored in the memory to implement the method provided in an embodiment of the present application.

[0028] A computer-readable storage medium is provided in an embodiment of the present application, and the computer-readable storage medium stores computer-executable instructions, which are configured to be executed by a processor to implement the method for target detection provided in an embodiment of the present application.

[0029] A computer program product is provided in an embodiment of the present application, and the computer program product includes computer-executable instructions, which are configured to be executed by a processor to implement the method for target detection provided in an embodiment of the present application.

[0030] The embodiments of the present application have the following beneficial effects: a closed set target detection model is used to perform closed set target detection processing on a target image to obtain a first target detection result. Then, an open set target detection model is used to perform open set target detection processing on the target image to obtain a second target detection result. The detection accuracy of the first target detection result is higher than that of the second target detection result. The closed set detection model can accurately identify the category of the target image, thereby improving the accuracy of the final target detection result. Meanwhile, the open set detection model can identify the category of all target objects in the target image, thereby ensuring that the final target detection result can include all target objects in the target image as much as possible. The first target detection result and the second target detection result are fused to obtain the target detection result of the target image. The first target detection result with higher accuracy and the second target detection result including as many target objects in the target image as possible are fused to obtain the target detection result, which improves the accuracy of target detection in the case of open set detection. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 FIG. 1 is a schematic diagram of an architecture of a system 100 for target detection provided in an embodiment of the present application;

[0032] Figure 2 is a structural schematic diagram of an electronic device 500 provided by an embodiment of the present application;

[0033] Figure 3A is a first flowchart of a target detection method provided by an embodiment of the present application;

[0034] Figure 3B is a second flowchart of a target detection method provided by an embodiment of the present application;

[0035] Figure 3C is a third flowchart of a target detection method provided by an embodiment of the present application;

[0036] Figure 3D is a fourth flowchart of a target detection method provided by an embodiment of the present application;

[0037] Figure 4 is a working principle diagram of a segmentation model provided by an embodiment of the present application;

[0038] Figure 5 is an implementation flowchart in an actual application scenario provided by an embodiment of the present application.

[0039] It should be noted that the above-mentioned "first", "second" are only used to distinguish different schemes, and do not represent the advantages or disadvantages of the schemes or the priority in the implementation process. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by a person of ordinary skill in the art without making creative labor fall within the scope of protection of the present application.

[0041] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0042] In the following description, the terms "first", "second", "third" are only used to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first", "second", "third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0044] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as commonly understood by one of ordinary skill in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0045] The relevant data collection process in the embodiments of the present application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of the laws and regulations and the personal information subject, carry out subsequent data use and processing.

[0046] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0047] 1) Open set detection: Open set detection refers to the process of detecting target objects using machine learning algorithms and computer vision techniques. In this process, open set detection mainly analyzes the environment around the target object and uses open set theory to identify the area or location of the target object. Specifically, open set detection technology can identify the difference between the target object and the surrounding background by analyzing the pixels or features in the image, thereby determining the location and bounding box of the target.

[0048] 2) Open set detection model: An open set detection model can identify unknown targets appearing in an image, not just limited to predefined categories. For example, models such as label-prompted object detection large models, deep learning-based prototype networks, autoencoders, and multi-task learning models can achieve open set detection.

[0049] 3) Closed set detection: Closed set detection refers to the process of detecting and identifying targets within a predefined set of categories. Closed set detection is a process of detecting targets within a known category range, i.e., the model can only identify categories that have been trained, and cannot identify targets appearing in the image that do not belong to the known categories.

[0050] 4) Closed-set detection model: The closed-set detection model refers to a target detection model used to identify a set of predefined categories. The closed-set detection model only focuses on these categories during training, and is expected to be able to identify and locate targets in these categories during testing. For example, Regions with CNN features (R-CNN), Mask Regions with CNN features (Mask R-CNN), and RetinaNet, etc.

[0051] Target detection is a very important cornerstone in the field of computer vision, and it has very many applications, such as autonomous driving, machine vision, security video, pedestrian detection, etc. In closed-set detection, Mask-RCNN and DETR have achieved very good results, but the capabilities of these algorithms are limited to the categories predefined during training. In the real world, the categories of various objects follow a long-tail distribution, and there are many rare and unusual categories. The closed-set target detector cannot meet the needs of different application scenarios.

[0052] Open-set detection is a very challenging problem in the field of image detection. Traditional target detection algorithms are essentially closed-set detection and are difficult to handle new object detection problems. In the early days of target detection, Generalized Novel Class Discovery (GNCD) can use semi-supervised and contrastive learning to achieve the recognition of new objects, but since this method cannot achieve the positioning of the target object, it is different from open-set detection.

[0053] With the development of Vision-Language Models (VLMs) such as CLIP, by integrating large models, the class name representation ability of large models is used to achieve open-set detection of target recognition. However, this method depends on a large amount of training data and computing resources, and the cost is extremely high.

[0054] The embodiments of the present application provide a target detection method, device, electronic equipment, computer readable storage medium and computer program product, which can improve the accuracy of target detection while ensuring open-set detection.

[0055] The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device, a smart device, a vehicle-mounted terminal, etc. Various types of terminals, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a server.

[0056] Referring to Figure 1 ,Figure 1 is an architecture diagram of a system 100 for target detection provided by an embodiment of the present application. To implement an application of supporting target detection, a terminal 400 connects a server 200 through a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0057] The terminal 400 is configured to acquire a target image for target detection, and then transmit the target image to the server 200 through the network 300.

[0058] The server 200 is configured to receive the target image transmitted by the terminal 400 through the network 300, then perform closed set target detection processing on the target image through a closed set target detection model to obtain a first target detection result, and perform open set target detection processing on the target image through an open set target detection model to obtain a second target detection result, wherein the detection accuracy of the closed set target detection model is higher than that of the open set target detection model, and finally fuse the first target detection result and the second target detection result to obtain a target detection result of the target image. After obtaining the target detection result, the server 200 can return the target detection result to the terminal 400 through the network 300.

[0059] The target detection method provided by the embodiment of the present application can be applied to an automatic driving scene, an image search scene, and a scene of a recommendation system. In the automatic driving scene, the terminal can collect surrounding images in the process of automatic driving as target images and transmit the target images to the server. The server performs closed set target detection processing on the target images through a closed set target detection model to obtain a first target detection result, and performs open set target detection processing on the target images through an open set target detection model to obtain a second target detection result, wherein the detection accuracy of the closed set target detection model is higher than that of the open set target detection model, and finally fuses the first target detection result and the second target detection result to obtain a target detection result of the target image. The server sends the recognized target detection result to the terminal, and the terminal uses the target detection result as a corresponding automatic driving instruction.

[0060] In an image search scenario, a user can input a target image requiring image search into a terminal, the terminal can transmit the target image to a server, the server performs closed set target detection processing on the target image by using a closed set target detection model to obtain a first target detection result, and performs open set target detection processing on the target image by using an open set target detection model to obtain a second target detection result, where the detection accuracy of the closed set target detection model is higher than that of the open set target detection model, and finally the first target detection result and the second target detection result are fused to obtain a target detection result of the target image, and the server sends the recognized target detection result to the terminal, and the terminal shows the target detection result to the user.

[0061] In a recommendation system, the terminal can obtain a commodity interface frequently browsed by a user as a target image with the permission of the user, then transmit the target image to the server, the server performs closed set target detection processing on the target image by using a closed set target detection model to obtain a first target detection result, and performs open set target detection processing on the target image by using an open set target detection model to obtain a second target detection result, where the detection accuracy of the closed set target detection model is higher than that of the open set target detection model, and finally the first target detection result and the second target detection result are fused to obtain a target detection result of the target image, and the server sends the recognized target detection result to the terminal, and the terminal sends the target detection result into a recommendation model to output a recommendation result by the recommendation model.

[0062] The electronic device for performing target detection provided in the embodiments of the present application can be various types of terminal devices or servers. In some embodiments, the server 200 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0063] Referring to Figure 2 , Figure 2 is a structural schematic diagram of an electronic device 500 provided in the embodiments of the present application, Figure 2The illustrated electronic device 500 includes at least one processor 510, memory 550, at least one network interface 520, and a user interface 530. The various components of electronic device 500 are coupled together by a bus system 540, which is configured to permit communication between the components. The bus system 540 can include a power bus, a control signal bus, and status signal bus, among others. For the sake of clarity, the various buses are illustrated in Figure 5 as the bus system 540. The processor 510 can be implemented as one or more integrated circuit chips designed to process information including program instructions and data. The processor 510 can be a general-purpose processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to process information including program instructions and data. Figure 2 The various buses are illustrated in Figure 5 as the bus system 540 for clarity.

[0064] The processor 510 can be an integrated circuit chip that has the processing capability to process signals, such as a general-purpose processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to process information including program instructions and data, where the general-purpose processor can be a microprocessor or any conventional processor.

[0065] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532 that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0066] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 550 optionally includes one or more storage devices remotely located from the processor 510.

[0067] The memory 550 includes volatile memory or non-volatile memory, and can include both volatile and non-volatile memory. Non-volatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 550 described in embodiments of the present application is intended to include any suitable type of memory.

[0068] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are described below by way of example.

[0069] The operating system 551 includes system programs for processing various basic system services and performing hardware-dependent tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks; The operating system 551 includes system programs for processing various basic system services and performing hardware-dependent tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0070] a network communication module 552 for communicating to other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB), etc.;

[0071] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating a peripheral device and displaying content and information);

[0072] an input processing module 554 for detecting and interpreting one or more user inputs or interactions from one or more input devices 532.

[0073] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in software, Figure 2 An object detection apparatus 555 stored in the memory 550 is shown, which can be in the form of software such as programs and plug-ins, including the following software modules: a closed set detection module 5551, an open set detection module 5552, and a fusion processing module 5553, which are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0074] In some other embodiments, the apparatus provided by the embodiments of the present application can be implemented in hardware, as an example, the apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the object detection method provided by the embodiments of the present application, for example, the processor in the form of a hardware decoding processor can use one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), or other electronic elements.

[0075] Next, the target detection method provided by the embodiments of the present application is described. As mentioned above, the electronic device implementing the target detection method of the embodiments of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeated in the following description.

[0076] Referring to Figure 3A , Figure 3A is a first flowchart of the target detection method provided by the embodiments of the present application, which will be described in combination with the steps shown in Figure 3A .

[0077] In step 101, the target image is subjected to closed set target detection processing by a closed set target detection model to obtain a first target detection result.

[0078] As an example, before implementing the target detection method provided by the embodiments of the present application, a target image used for target detection needs to be acquired. The target image can be acquired by a terminal acquisition device, uploaded by a user, or obtained from a database. The acquisition method of the target image and the source of the target image can be selected according to actual conditions, and are not specifically limited here.

[0079] As an example, after the target image is acquired, the target image can be subjected to closed set target detection processing by a closed set target detection model to obtain a first target detection result.

[0080] As an example, closed set detection refers to a process of target detection and recognition in a predefined category set. Closed set detection is target detection within a known category range, i.e., the model can only recognize categories that have been trained, and cannot recognize targets that appear in the image but do not belong to the known categories.

[0081] As an example, a closed set detection model refers to a target detection model used to recognize a set of predefined categories. The closed set detection model only focuses on these categories during training, and is expected to recognize and locate targets in these categories during testing. The closed set detection model can include a region convolutional neural network (Regions with CNN features, R-CNN), a mask region convolutional neural network (Mask Regions with CNN features, Mask R-CNN), and a retina network (RetinaNet), etc.

[0082] In step 102, the target image is subjected to open set target detection processing by an open set target detection model to obtain a second target detection result, wherein the detection accuracy of the first target detection result is higher than the detection accuracy of the second target detection result

[0083] As an example, open set detection refers to a process of detecting target objects using machine learning algorithms and computer vision techniques. In this process, open set detection mainly analyzes the environment around the target object and uses open set theory to identify the area or location of the target object. Specifically, open set detection technology can identify the difference between the target object and the surrounding background by analyzing the pixels or features in the image, thereby determining the location and bounding box of the target.

[0084] As an example, the open set detection model can identify unknown targets appearing in the image, not just limited to predefined categories. For example, the open set detection model can be a label prompt target detection large model, a deep learning-based prototype network, an autoencoder, and a multi-task learning model.

[0085] In step 103, the first target detection result and the second target detection result are fused to obtain the target detection result of the target image.

[0086] In some embodiments, the fusion of the first target detection result and the second target detection result in step 103 to obtain the target detection result of the target image can be implemented by steps 1031 to 1032 as shown in the following: Figure 3B

[0087] In step 1031, the first target detection result and the second target detection result are spliced to obtain a target detection splicing result.

[0088] In some embodiments, the first target detection result includes a first detection box with a description parameter and a second detection box without a description parameter, and the second target detection result includes a third detection box with a description parameter; the target detection splicing result includes a detection box splicing result and a parameter splicing result; the splicing of the first target detection result and the second target detection result in step 1031 to obtain the target detection splicing result can be implemented by steps 10311 to 10312 as shown in the following: Figure 3C

[0089] In step 10311, the first image block corresponding to the first detection box, the second image block corresponding to the second detection box, and the third image block corresponding to the third detection box are spliced to obtain a detection box splicing result.

[0090] ​​As an example, since the closed-set object detection model is trained based on limited training samples, there may be objects in the target image that cannot be recognized by the closed-set object detection model in the process of object detection by the closed-set object detection model. Therefore, the first object detection result includes known detection results that can be accurately detected by the closed-set object detection model and unknown detection results that cannot be accurately detected by the closed-set object detection model. The known detection results include first detection boxes with description parameters, and the unknown detection results include second detection boxes without description parameters.

[0091] As an example, the description parameters of the detection box can include at least one of the confidence of the detection box and the category of the detection box. The confidence of the detection box can be used to measure the accuracy of the detection box positioned by the target detection model, and can also be used to measure the recognition accuracy of the target object in the detection box by the target detection model. The actual meaning of the confidence of the detection box can be any one of the accuracy of the detection box positioned by the target detection model and the recognition accuracy of the target object in the detection box by the target detection model, or the confidence of the detection box obtained by combining the accuracy of the detection box positioned by the target detection model and the recognition accuracy of the target object in the detection box by the target detection model. The actual meaning of the confidence of the detection box can be selected according to actual conditions, which is not specifically limited here.

[0092] The category of the detection box can be the image category of the target object in the detection box, i.e., the label category of the target object, such as a table, a chair, a clock, etc., or the edge box category of the detection box, such as a quadrilateral box, a circular box, a polygonal box, and other shape boxes. The actual meaning of the category of the detection box can be one of the label category of the target object and the edge box category of the detection box, or the category of the detection box that includes both the label category of the target object and the edge box category of the detection box. The actual meaning of the category of the detection box can be selected according to actual conditions, which is not specifically limited here.

[0093] As an example, the first image block corresponding to the first detection box includes image block A and image block B, the second image block corresponding to the second detection box includes image block C and image block D, and the third image block corresponding to the third detection box includes image block E and image block F. The detection box splicing result obtained by splicing the first image block, the second image block and the third image block can be [image block A, image block B, image block C, image block D, image block E and image block F].

[0094] As an example, in actual application, there can be image blocks that overlap between the first image block, the second image block and the third image block, that is, the first image block includes image block A and image block B, the second image block includes image block B and image block C, and the third image block includes image block C and image block D. At this time, the image blocks can be selected according to the confidence of the image blocks, for example, the confidence of image block B in the first image block is 0.9, and the confidence of image block B in the second image block is 0.7, so that image block B included in the detection frame splicing result obtained by splicing is image block B included in the first image block. Similarly, if the confidence of image block C in the second image block is 0.8, and the confidence of image block C in the third image block is 0.7, then image block C included in the detection frame splicing result obtained by splicing is image block C included in the second image block.

[0095] In step 10312, the description parameters of the first detection frame and the description parameters of the third detection frame are spliced to obtain a parameter splicing result.

[0096] As an example, the description parameters of the first detection frame are description parameters A, and the description parameters of the third detection frame are description parameters C, so that the parameter splicing result can be [description parameters A, description parameters C]

[0097] In some embodiments, after performing the closed set target detection model in step 101 above, the target image is detected by the closed set target detection model to obtain the first target detection result, the following technical solution can also be performed: the second detection frame is subjected to similarity matching processing to obtain the description parameters of the second detection frame. After performing the above technical solution, the description parameters of the first detection frame and the description parameters of the third detection frame in step 10312 are spliced to obtain a parameter splicing result, which can be realized by the following technical solution: the description parameters of the first detection frame, the description parameters of the second detection frame and the description parameters of the third detection frame are spliced to obtain a parameter splicing result.

[0098] As an example, the description parameters of the first detection frame are description parameters A, the description parameters of the second detection frame are description parameters B, and the description parameters of the third detection frame are description parameters C, so that the parameter splicing result can be [description parameters A, description parameters B, description parameters C].

[0099] In some embodiments, the description parameter of the second detection box includes an image category of the second detection box and a detection box confidence of the second detection box; and the similarity matching processing of the second detection box to obtain the description parameter of the second detection box can be implemented by the following technical solution: obtaining a category feature of a candidate detection category; performing image feature extraction processing on a second image block corresponding to the second detection box to obtain an image feature of the second image block corresponding to the second detection box; determining a feature similarity between the image feature and the category feature; and if the feature similarity is higher than a similarity threshold, taking the candidate detection category corresponding to the category feature as the image category of the second detection box, and taking the feature similarity as the detection box confidence of the second detection box.

[0100] As an example, the category feature of the candidate detection category and the image feature of the second image block corresponding to the second detection box can be obtained first, and then the similarity between the category feature and the image feature is determined to determine the category of the second detection box.

[0101] As an example, the candidate detection category can be a category that can exist in a target image set in advance, for example, the candidate detection category can include a clock, a table, a chair, a telephone, etc.

[0102] In some embodiments, the above-mentioned obtaining of the category feature of the candidate detection category can be implemented by steps 10312A to 10312D as shown in the following. Figure 3D

[0103] In step 10312A, a keyword of the candidate detection category is obtained.

[0104] As an example, after the candidate detection category is determined, the keyword of each candidate detection category can be determined, wherein the keyword of the candidate detection category is a keyword related to the candidate detection category.

[0105] As an example, the candidate detection category A is a cat, and the keyword of the candidate detection category A can be a short-haired cat. The candidate detection category B is a table, and the keyword of the candidate detection category B can be a table.

[0106] In step 10312B, a text description of the keyword is generated based on the keyword of the candidate detection category.

[0107] As an example, in order to improve the accuracy of the corresponding category feature of the subsequent candidate detection category, each candidate category can include multiple keywords, and the technical solution when each candidate category includes multiple keywords is introduced as follows.

[0108] ​In some embodiments, the text description of the keyword includes a first text description and a second text description, and the generation of the text description of the keyword based on the keyword of the candidate detection category in step 10312B can be implemented by the following technical solutions: synonym expansion is performed on the keyword to obtain a synonym keyword of the keyword; and text generation processing is performed on the keyword and the synonym keyword to obtain the first text description including the keyword and the second text description including the synonym keyword.

[0109] As an example, after determining the keyword of the candidate detection category, synonym expansion can be performed on the keyword to obtain a synonym keyword of the keyword.

[0110] As an example, the keyword A is a short-haired cat, and the synonym keyword B of the keyword A can be a long-haired cat. The keyword C is a chair, and the synonym keyword C of the keyword C can be a deck chair.

[0111] As an example, synonym expansion can be performed based on a large language model, that is, the keyword is taken as an input of the large language model, the generated synonym is taken as a prompt word of the large language model, and the synonym keyword of the keyword is output by the large language model.

[0112] As an example, the large language model is a machine learning model used for generating natural language text, which can process languages with broad meanings and complex structures, such as dialogue systems, automatic writing, intelligent customer service, etc. The large language model usually uses deep learning techniques, especially recurrent neural network (RNN), long short-term memory network (LSTM) or gated recurrent unit (GRU) and other recurrent neural network structures, and transformer (Transformer) model. The large language model can process a large amount of text data, and by learning the grammar, semantics and context information of the language, it can generate natural and fluent text. The large language model usually needs a large amount of labeled data for training, and needs high-performance computers or cloud computing resources to process large-scale data.

[0113] As an example, after obtaining the keyword and the synonym keyword, the first text description can be generated based on the obtained keyword, and the second text description can be generated based on the synonym keyword.

[0114] As an example, the keyword A is a short-haired cat, and the first text description can be There is a short-haired cat here. The synonym keyword of the keyword A is a long-haired cat, and the second text description can be There is a long-haired cat here.

[0115] It should be noted that in the above examples, the first text description and the second text description have the same sentence structure, and only the keyword is different. In actual application, the first text description and the second text description can have the same sentence structure, or can have different sentence structures.

[0116] As an example, the keyword A is a short-haired cat, and the first text description can be There is a short-haired cat at the door. The synonym keyword of keyword A is a long-haired cat, and the second text description can be This long-haired cat is really beautiful.

[0117] As an example, after obtaining the keyword and the synonym keyword, the keyword and the synonym keyword can be input into a large language model to generate a text description as a prompt word, and the large language model generates a first text description corresponding to the keyword and a second text description corresponding to the synonym keyword.

[0118] In the above manner, the number of keywords included in each category can be increased, and the accuracy of subsequent category features is improved.

[0119] In step 10312C, text feature extraction processing is performed on the text description of the keyword to obtain the text feature of the text description of the keyword.

[0120] In some embodiments, the text feature extraction processing on the text description of the keyword in step 10312C to obtain the text feature of the text description of the keyword can be realized by the following technical solution: performing word segmentation processing on the text description to obtain the lexical units included in the text description; performing feature embedding processing on the lexical units to obtain the unit embedding features of the lexical units; performing splicing processing on the unit embedding features of the plurality of lexical units to obtain the text feature of the keyword.

[0121] As an example, the text description information can be first processed by word segmentation to obtain the lexical units included in the text description. This process usually includes segmenting words, punctuation marks, numbers, etc. in the text, and removing some noise or redundant information.

[0122] As an example, the text description information A is There is a cat sitting at the door. After word segmentation processing on the text description information A, the lexical units of the text description information A can be "has", "one", "cat", "at", "door", and "sitting".

[0123] As an example, after obtaining the lexical units of the text description information, the lexical units can be processed by feature embedding to obtain the unit embedding features of the lexical units. That is, the lexical units are mapped to a vector space.

[0124] As an example, the feature embedding processing on the lexical units can use a pre-trained word vector model (such as Word2Vec, etc.). These models map each lexical unit to a fixed-length vector, so that the vector representation of each lexical unit can capture its semantic and contextual information.

[0125] As an example, a word vector model (Word2Vec) is a word embedding model that can learn distributed representations of words, i.e., map each word to a continuous vector space, through training. Word2Vec can capture semantic relationships between words and provide an effective feature representation method for natural language processing tasks.

[0126] As an example, the lexical units of the text description information A can be "has", "a", "cat", "at", "the doorway", and "sitting". Through feature embedding processing, the unit embedding features corresponding to the text description information A can be obtained as token1 (unit embedding feature corresponding to "has"), token2 (unit embedding feature corresponding to "a"), token3 (unit embedding feature corresponding to "cat"), token4 (unit embedding feature corresponding to "at"), token5 (unit embedding feature corresponding to "the doorway"), and token6 (unit embedding feature corresponding to "sitting").

[0127] As an example, after obtaining the unit embedding features, the unit embedding features can be concatenated to obtain the text features of the keyword.

[0128] As an example, the unit embedding features corresponding to the text description information A are token1 (unit embedding feature corresponding to "has"), token2 (unit embedding feature corresponding to "a"), token3 (unit embedding feature corresponding to "cat"), token4 (unit embedding feature corresponding to "at"), token5 (unit embedding feature corresponding to "the doorway"), and token6 (unit embedding feature corresponding to "sitting"). The unit embedding features of the text description information A can be concatenated as token1, token2, token3, token4, token5, and token6, and the features obtained after concatenation can be used as the text features of the keyword.

[0129] In the above manner, the accuracy of the extracted text features can be improved, and the accuracy of the subsequent determination of the category features is also improved.

[0130] In step 10312D, based on the text features of the text description of the keyword, the category features of the candidate detection category are determined.

[0131] In some embodiments, the text feature of the keyword-based text description in step 10312D, and the class feature of the candidate detection class can be realized by the following technical solutions: if the keyword has multiple text descriptions, the text features of the multiple text descriptions are fused to obtain the keyword feature of the keyword; if the keyword has one text description, the text feature of the text description is determined as the keyword feature of the keyword; if the keyword has a synonymous keyword, the keyword feature of the keyword and the keyword feature of the synonymous keyword are fused to obtain the class feature of the candidate detection class; if the keyword does not have a synonymous keyword, the keyword feature of the keyword is determined as the class feature of the candidate detection class.

[0132] As an example, when a keyword only has one text description, the text feature of the text description can be determined as the keyword feature.

[0133] As an example, when a keyword has multiple text descriptions, the text features of the multiple text descriptions of the keyword need to be fused.

[0134] As an example, the keyword A is a short-haired cat, the text description B corresponding to the keyword A is here is a short-haired cat, and the text description C corresponding to the keyword A is this short-haired cat is really beautiful. At this time, the text feature D corresponding to the text description B and the text feature E corresponding to the text description C can be fused to obtain the keyword feature of the keyword A.

[0135] As an example, the way of fusing the text features can be to determine the average text feature between the text features corresponding to the multiple text descriptions, and the obtained average text feature is taken as the keyword feature of the keyword corresponding to the text description.

[0136] As an example, if the keyword has a synonymous keyword, the keyword feature of the keyword and the keyword feature of the synonymous keyword can be fused to obtain the class feature of the candidate detection class.

[0137] As an example, after the keyword feature corresponding to the keyword is determined, the keyword feature of the synonymous keyword corresponding to the keyword can be determined based on the same method. Then the keyword features corresponding to the keywords and the keyword features corresponding to the synonymous keywords which belong to one candidate detection class can be fused to obtain the class feature of the candidate detection class.

[0138] As an example, after the keyword feature corresponding to the keyword and the keyword feature of the synonymous keyword are obtained, the keyword feature of the keyword and the keyword feature of the synonymous keyword can be summed, and then the sum result is averaged to obtain the average keyword feature, which can be taken as the class feature of the candidate detection class.

[0139] As an example, if the keyword does not have a synonym keyword, the keyword feature of the keyword can be determined as the class feature of the candidate detection class.

[0140] As an example, by the above manner, more text features and keyword features can be fused, the accuracy of the obtained class feature is improved, and the accuracy of subsequent target detection is further improved.

[0141] As an example, the process of performing image feature extraction processing on the second image block corresponding to the obtained image block can be implemented by using a contrastive language-image pre-training (CLIP) model.

[0142] The CLIP is a model pre-trained using a large number of images and texts, which learns to associate image content with natural language descriptions. Specifically, the CLIP model uses a large number of pictures and corresponding descriptive texts collected from the Internet. These texts can be the title, label or other description information of the picture. The CLIP model includes two encoders, one is an image encoder and the other is a text encoder. The image encoder usually uses a convolutional neural network to extract the features of the image, while the text encoder uses a transformer architecture to extract the features of the text. The image encoder processes the image to generate a feature representation, and the text encoder processes the text to also generate a feature representation. These two feature representations are the internal representations of the image and the text in the model, respectively. The CLIP trains the model by comparing the feature representations of the image and the text. The CLIP generates a positive sample for each image-text pair, i.e., the image and its corresponding text, and generates multiple negative samples for each image-text pair, i.e., the image and other unmatched texts. In the pre-training phase, the CLIP model is trained on a large-scale image-text pair data, and the learned features can be used for subsequent tasks such as image classification, target detection, etc. In a specific downstream task, the CLIP model can be fine-tuned, i.e., using less data to adjust the parameters of the model to better adapt to the specific task requirements.

[0143] As an example, in addition to using the CLIP model for image feature extraction processing, neural networks such as convolutional neural networks, recurrent neural networks, generators in generative adversarial networks, autoencoders, and graph convolutional neural networks can also be used for image feature extraction processing. The neural network used for image feature extraction processing on the second image block can be selected according to actual conditions, and is not specifically limited here.

[0144] As an example, after determining the image feature and the category feature, the feature similarity between the image feature and the category feature can be determined. Specifically, the cosine similarity between the image feature and the category feature can be calculated by the CLIP model described above, and then the feature similarity between the image feature and the category feature is obtained.

[0145] Specifically, the cosine similarity between the image feature and the category feature can be determined by using the following formula:

[0146]

[0147] In formula (1), a is the image feature, b is the category feature, ‖a‖ is the length of the image feature, ‖b‖ is the length of the category feature, and COS(a, b) is the cosine similarity between the image feature and the category feature.

[0148] According to formula (1), the greater the cosine similarity, the less similar the image feature and the category feature are, so in actual application, the reciprocal of the cosine similarity can be used as the feature similarity between the image feature and the category feature. The above is only one way to obtain the cosine similarity, and in actual application, the reciprocal of the cosine similarity can also be used as the feature similarity between the image feature and the category feature. The processing of the cosine similarity can be selected according to the actual situation, and is not specifically limited here.

[0149] As an example, the above is only a way to determine the feature similarity between the image feature and the category feature by using the cosine similarity, and in actual application, the feature similarity between the image feature and the category feature can also be determined by using the Euclidean distance between the image feature and the category feature. The way to determine the feature similarity between the image feature and the category feature can be selected according to the actual situation, and is not specifically limited here.

[0150] As an example, after obtaining the feature similarity, if the feature similarity is higher than the similarity threshold, the candidate detection category corresponding to the category feature can be used as the category of the second detection box.

[0151] As an example, the similarity threshold is 0.8, the feature similarity between the image feature B corresponding to the second detection box A and the category feature C of the candidate detection category of a chair is 0.85, and the chair can be used as the category of the second detection box A.

[0152] As an example, since the feature similarity reflects the similarity degree of the target object in the second detection box and the candidate detection category, that is, the higher the feature similarity, the more likely the target object in the second detection box belongs to the candidate detection category. Therefore, after determining the category of the second detection box, the feature similarity between the corresponding image feature of the second detection box and the category feature of the category can be taken as the detection box confidence of the second detection box.

[0153] In step 1032, based on the target detection splicing result, a target detection result of the target image is determined.

[0154] In some embodiments, the description parameter includes a detection box confidence; and the determination of the target detection result of the target image based on the target detection splicing result in step 1032 can be implemented by the following technical solution: performing target image block-based image segmentation processing on the target image to obtain at least one mask belonging to a target image block and a mask confidence of each mask, wherein the target image block is any image block from the detection box splicing result; generating a bounding box corresponding to the first mask in the target image as a target detection box, wherein the first mask is any one of the at least one mask; determining a target confidence of the target detection box based on the detection box confidence included in the parameter splicing result and the first mask confidence; and generating the target detection result based on the target detection box and the target confidence.

[0155] As an example, the target image block and the target image can be input into a Segment Anything Model (SAM) model, and the first mask and the confidence of the first mask can be output by the SAM model.

[0156] As an example, the SAM model is designed to implement an image segmentation task. The design concept of the SAM model is to have a wide generalization ability, and can generate accurate segmentation masks for objects in any image, even for objects and image types that have not been directly encountered in the training process. This ability is called zero-shot transfer in the field of deep learning, which is an important feature of the SAM model.

[0157] The SAM model is composed of an encoder and a decoder, and the basic principle is to process the input image through the encoder to capture the global features and context information in the image. The decoder part is responsible for generating accurate segmentation masks for each object in the image according to the information provided by the encoder.

[0158] As an example, the following will be described in combination with Figure 4 The process of determining the target detection result of the target image will be introduced. Figure 4 is a working principle diagram of the segmentation model provided by the embodiments of the present application.

[0159] Firstly, the target image is input to the encoding layer, and then the image features of the target image are obtained through convolution processing. Then, the target image block is input to the mask decoder as a prompt word to obtain the first mask of the output target image block (as shown in 401 in FIG. 4) and the mask confidence corresponding to each mask. Figure 4

[0160] As an example, after obtaining the first mask of the target image block, a bounding box of the first mask can be generated, as shown in 402 in FIG. 4, and the bounding box is taken as the target detection box. Figure 4

[0161] As an example, based on the detection box confidence and the mask confidence included in the parameter splicing result, the target confidence of the target detection box is determined. The process of determining the target confidence is described in detail below.

[0162] In some embodiments, the above determination of the target confidence of the target detection box based on the detection box confidence and the mask confidence of the first mask included in the parameter splicing result can be realized through the following technical solution: the detection box confidence and the mask confidence of the first mask are respectively subjected to standardization processing to obtain a standardized detection box confidence and a standardized mask confidence; the standardized detection box confidence and the standardized mask confidence having a corresponding relationship are multiplied to obtain the target confidence, wherein the corresponding relationship represents that the first mask corresponding to the standardized mask confidence is within the detection box corresponding to the standardized detection box confidence.

[0163] As an example, after obtaining the mask confidence and the detection box confidence, the mask confidence and the detection box confidence can be subjected to standardization processing by using the following formula (2).

[0164]

[0165] In formula (2), x' is the standardization processing result, x is the standardized sequence, min(x) is the minimum value in the standardized sequence, and max(x) is the maximum value in the standardized sequence.

[0166] As an example, the detection box confidence can be taken as a standardized sequence and substituted into the above formula (2) to obtain the standardized detection box confidence. Similarly, the mask confidence can be substituted into the above formula (2) to obtain the standardized mask confidence. It should be noted that in actual application, the detection box confidence and the mask confidence can also be fused into a standardized sequence and substituted into the above formula (2) to obtain the standardized detection box confidence and the standardized mask confidence after standardization.

[0167] ​​As an example, after obtaining the standardized bounding box confidence and the standardized mask confidence, the standardized bounding box confidence and the standardized mask confidence with the corresponding relationship can be multiplied to obtain the target confidence.

[0168] As an example, the bounding box A includes the image block A, and the mask B is obtained based on the image block A, so the bounding box A has a corresponding relationship with the mask B. If the standardized bounding box confidence corresponding to the bounding box A is 0.7, and the standardized mask confidence corresponding to the mask B is 0.8, then the target confidence is 0.7*0.8, which is 0.56.

[0169] In the above manner, the accuracy of the obtained bounding box confidence can be improved, and the accuracy of target detection is further improved.

[0170] As an example, after obtaining the target confidence, the target detection result can be generated based on the target confidence and the target bounding box.

[0171] As an example, in the process of generating the target detection result, the bounding box class can also be taken as part of the target detection result.

[0172] Next, an example application of the embodiment of the present application in an actual application scenario will be described.

[0173] First, the embodiment of the present application will be described in combination with Figure 5 The overall process of the embodiment of the present application in an actual application scenario will be introduced, Figure 5 which is an implementation flowchart of the embodiment of the present application in an actual application scenario.

[0174] First, a target image is input, and then the target image is subjected to target detection through two pre-trained models, which are a closed set detection model (Mask-RCNN) and an open set detection model (GroundingDINO). The target image can be detected by the open set detection model to obtain a second target detection result, which includes a bounding box B GD , a score S GD of the bounding box, and a label L GD of the bounding box. The output of the closed set detection model has two parts, one of which is a first target detection result, which includes a bounding box B KN , a score S KN of the bounding box, and a label L KN of the bounding box, and the other is an unknown detection result, which includes a bounding box B UKN , an unknown bounding box score S UKNand the unknown detection frame category L UKN The first detection result, the second detection result, and the unknown detection result are then respectively input to a synonym embedding generator (SEG) and a refinement module (Refinement) for subsequent calculation, and finally a target detection result as shown in Figure 5 is obtained.

[0175] The working principle of the SEG will be described in detail below.

[0176] Firstly, the SEG can define a template set, which can include templates of multiple different categories (categories of targets), such as Figure 5 As shown in the prompt word 1 and the prompt word 2 are templates belonging to the same category.

[0177] Then, the SEG can define multiple keywords under each category, which are synonyms of each other. The definition process is to first determine a keyword, and then determine the synonyms of the keyword as multiple synonymous keywords under the category, such as Figure 5 “clock” and “wall clock” in

[0178] Then, in each template, a set of text prompts is generated according to the respective keywords. For example, according to the “clock” keyword, “there is a clock in the scene” and “there is a wall clock on a wall in the scene” are generated.

[0179] Then, in each template, the CLIP (Contrastive Language-Image Pre-Training) is used to tokenize and encode each text description to obtain the text feature (embedding) corresponding to each text description.

[0180] Next, the text embedding in each template is normalized, and the average feature vector is calculated as the text feature of the template.

[0181] Then, the text features of different templates belonging to synonyms are summed up, and the average feature is calculated to obtain the final feature representation T1 of the category.

[0182] Finally, the cosine example calculation method of the CLIP model is used to obtain the category label of the image.

[0183] The input of SEG is the unknown object detected by Mask-RCNN, expressed in small image blocks. The output is the category and probability score value of the image block calculated by CLIP, that is, the second detection box included in the unknown detection result and the description parameters of the second detection box.

[0184] The role of SEG is to obtain more rich semantic information by synonym expansion on the category, which can be used as a paradigm of text description generation.

[0185] The working principle of Refinement is introduced below.

[0186] First, the obtained B GD , S GD , L GD , B KN , S KN , L KN , B UKN , S UKN and L UKN are spliced to obtain the complete detection result, that is, B GD , B KN and B UKN are spliced to obtain B CB , S GD , S KN and S UKN are spliced to obtain S CB , and L GD , L KN and L UKN are spliced to obtain L CB .

[0187] Then, the structural analysis model (Structural Analysis Model, SAM) is used to refine the detection box in the detection result, and the input to the SAM model is the target image and the detection box B CB (taking the detection box B CB as a prompt word), and the output is two values, one of which is the mask corresponding to the target object, and then the bounding box of each mask is determined, and the bounding box of the mask is taken as the final detection box B SAM , and the other output result is the score S SAM corresponding to each mask.

[0188] Then, the score refinement module (Score Refinement Module, SRM) is used to fuse the two scores (S CB and S SAM ), and the final score S is output. S CB and S SAMThe fusion manner can be: performing standardization on S CB and S SAM , and then multiplying the elements of one position to obtain the refined score, i.e., the confidence score of each detection frame.

[0189] Finally, B SAM , S and L CB are taken as the target detection result of the target image.

[0190] The refinement module can effectively combine the detection results of the closed set detection model and the open set detection model, utilize the advantages of the two detection models, and refine the detection frame and the detection frame score through a preset method.

[0191] The target detection method provided in the embodiments of the present application can expand the detection range of an existing detector to an open set through an innovative cooperation mechanism, without the need for training. The entire architecture of the target detection method provided in the embodiments of the present application is modularly designed, and can seamlessly integrate other detectors. The proposed cooperation mechanism of the target detection method provided in the embodiments of the present application can be applied not only to target detection, but also to image segmentation tasks.

[0192] It can be understood that, in the embodiments of the present application, user information and other related data are involved, and when the embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions.

[0193] The following continues to describe an exemplary structure of the implementation of the target detection apparatus 555 provided in the embodiments of the present application as a software module. In some embodiments, as shown in FIG. 5B, the software module stored in the target detection apparatus 555 of the memory 550 can include: Figure 2 The closed set detection module 5551 is configured to perform closed set target detection processing on the target image through a closed set target detection model to obtain a first target detection result.

[0194] The open set detection module 5552 is configured to perform open set target detection processing on the target image through an open set target detection model to obtain a second target detection result, wherein the detection accuracy of the first target detection result is higher than that of the second target detection result.

[0195] The fusion processing module 5553 is configured to perform fusion processing on the first target detection result and the second target detection result to obtain a target detection result of the target image.

[0196]

[0197] ​In some embodiments, the fusion processing module 5553 is further configured to perform splicing processing on the first target detection result and the second target detection result to obtain a target detection splicing result; and determine the target detection result of the target image based on the target detection splicing result.

[0198] In some embodiments, the fusion processing module 5553 is further configured to perform splicing processing on the first image block corresponding to the first detection box, the second image block corresponding to the second detection box, and the third image block corresponding to the third detection box to obtain a detection box splicing result; and perform splicing processing on the description parameters of the first detection box and the description parameters of the third detection box to obtain a parameter splicing result.

[0199] In some embodiments, the fusion processing module 5553 is further configured to perform similarity matching processing on the second detection box to obtain the description parameters of the second detection box; and perform splicing processing on the description parameters of the first detection box, the description parameters of the second detection box, and the description parameters of the third detection box to obtain a parameter splicing result.

[0200] In some embodiments, the fusion processing module 5553 is further configured to obtain a class feature of a candidate detection class; perform image feature extraction processing on the second image block corresponding to the second detection box to obtain an image feature of the second image block corresponding to the second detection box; determine a feature similarity between the image feature and the class feature; if the feature similarity is higher than a similarity threshold, take the candidate detection class corresponding to the class feature as an image class of the second detection box, and take the feature similarity as a detection box confidence of the second detection box.

[0201] In some embodiments, the fusion processing module 5553 is further configured to obtain a keyword of the candidate detection class; generate a text description of the keyword based on the keyword of the candidate detection class; perform text feature extraction processing on the text description of the keyword to obtain a text feature of the text description of the keyword; and determine a class feature of the candidate detection class based on the text feature of the text description of the keyword.

[0202] In some embodiments, the fusion processing module 5553 is further configured to perform synonym expansion on the keyword to obtain a synonymous keyword of the keyword; and perform text generation processing on the keyword and the synonymous keyword to obtain a first text description including the keyword and a second text description including the synonymous keyword.

[0203] In some embodiments, the fusion processing module 5553 is further configured to perform word segmentation processing on the text description to obtain a word unit included in the text description, perform feature embedding processing on the word unit to obtain a unit embedding feature of the word unit, and perform concatenation processing on unit embedding features of a plurality of the word units to obtain a text feature of the keyword.

[0204] In some embodiments, the fusion processing module 5553 is further configured to, if the keyword has a plurality of the text descriptions, perform fusion processing on text features of the plurality of the text descriptions to obtain a keyword feature of the keyword, if the keyword has one of the text descriptions, determine a text feature of the text description as the keyword feature of the keyword, if the keyword has a synonymous keyword, perform fusion processing on keyword features of the keyword and the synonymous keyword to obtain a class feature of the candidate detection class, and if the keyword does not have a synonymous keyword, determine the keyword feature of the keyword as the class feature of the candidate detection class.

[0205] In some embodiments, the fusion processing module 5553 is further configured to perform target image block-based image segmentation processing on the target image to obtain at least one mask belonging to a target image block and a mask confidence of each of the masks, where the target image block is any image block from the detection frame splicing result, generate a bounding box corresponding to a first mask in the target image as a target detection frame, where the first mask is any one of the at least one mask, determine a target confidence of the target detection frame based on a detection frame confidence included in the parameter splicing result and a mask confidence of the first mask, and generate the target detection result based on the target detection frame and the target confidence.

[0206] In some embodiments, the fusion processing module 5553 is further configured to perform standardization processing on the detection frame confidence and the mask confidence of the first mask, respectively, to obtain a standardized detection frame confidence and a standardized mask confidence, and multiply the standardized detection frame confidence and the standardized mask confidence having a corresponding relationship to obtain the target confidence, where the corresponding relationship indicates that the first mask corresponding to the standardized mask confidence is in the detection frame corresponding to the standardized detection frame confidence.

[0207] The embodiment of the present application provides a computer program product, which comprises a computer program or computer executable instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the target detection method provided in the embodiment of the present application.

[0208] The embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions or computer programs. When the computer executable instructions or computer programs are executed by the processor, the processor executes the target detection method provided in the embodiment of the present application, for example, the target detection method shown in the embodiment of the present application. Figure 3A

[0209] In some embodiments, the computer readable storage medium can be RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM memory, etc. It can also be various devices including one or any combination of the above storage.

[0210] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0211] As an example, the computer executable instructions can but not necessarily correspond to files in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code parts).

[0212] As an example, the computer executable instructions can be deployed to execute on one electronic device, or on multiple electronic devices located in one place, or on multiple electronic devices distributed in multiple places and interconnected through a communication network.

[0213] In summary, the embodiment of the present application can achieve the following beneficial effects:

[0214] ​The target image is subjected to closed set target detection processing through the closed set target detection model to obtain a first target detection result. Then the target image is subjected to open set target detection processing through the open set target detection model to obtain a second target detection result. The detection accuracy of the first target detection result is higher than that of the second target detection result. The closed set detection model can accurately identify the category of the target image, thereby improving the accuracy of the final target detection result. Meanwhile, the open set detection model can identify the category of all target objects in the target image, thereby ensuring that the final target detection result can include all target objects in the target image as much as possible. The first target detection result and the second target detection result are subjected to fusion processing to obtain the target detection result of the target image. The first target detection result with higher accuracy and the second target detection result including as many target objects in the target image as possible are subjected to fusion processing, and the obtained target detection result improves the accuracy of target detection in the case of open set detection.

[0215] The above merely illustrates the embodiments of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A method for target detection, characterized in that, The method includes: The target image is processed by a closed-set target detection model to obtain the first target detection result. The target image is processed by an open set target detection model to obtain a second target detection result. The detection accuracy of the first target detection result is higher than that of the second target detection result. The first target detection result and the second target detection result are fused to obtain the target detection result of the target image.

2. The method according to claim 1, characterized in that, The step of fusing the first target detection result and the second target detection result to obtain the target detection result of the target image includes: The first target detection result and the second target detection result are stitched together to obtain the target detection stitching result; Based on the target detection stitching results, the target detection result of the target image is determined.

3. The method according to claim 2, characterized in that, The first target detection result includes a first detection box with descriptive parameters and a second detection box without descriptive parameters; the second target detection result includes a third detection box with descriptive parameters; the target detection stitching result includes a detection box stitching result and a parameter stitching result. The step of stitching together the first target detection result and the second target detection result to obtain the target detection stitching result includes: The first image block corresponding to the first detection box, the second image block corresponding to the second detection box, and the third image block corresponding to the third detection box are stitched together to obtain the detection box stitching result. The description parameters of the first detection box and the description parameters of the third detection box are concatenated to obtain the parameter concatenation result.

4. The method according to claim 3, characterized in that, After performing closed-set target detection processing on the target image using the closed-set target detection model to obtain the first target detection result, the method further includes: The second detection box is subjected to similarity matching processing to obtain the description parameters of the second detection box; The process of concatenating the description parameters of the first detection frame and the description parameters of the third detection frame to obtain the parameter concatenation result includes: The description parameters of the first detection box, the second detection box, and the third detection box are concatenated to obtain the parameter concatenation result.

5. The method according to claim 4, characterized in that, The description parameters of the second detection box include the image category of the second detection box and the detection box confidence of the second detection box; The similarity matching process performed on the second detection box to obtain the description parameters of the second detection box includes: Obtain the category features of candidate detection categories; Image feature extraction processing is performed on the second image block corresponding to the second detection box to obtain the image features of the second image block corresponding to the second detection box; Determine the feature similarity between the image features and the category features; If the feature similarity is higher than the similarity threshold, then the candidate detection category corresponding to the category feature is taken as the image category of the second detection box, and the feature similarity is taken as the detection box confidence of the second detection box.

6. The method according to claim 5, characterized in that, The acquisition of category features for candidate detection categories includes: Obtain the keywords of the candidate detection categories; Based on the keywords of the candidate detection categories, generate a text description of the keywords; Text feature extraction processing is performed on the text description of the keywords to obtain the text features of the text description of the keywords; Based on the textual features of the textual description of the keywords, the category features of the candidate detection categories are determined.

7. The method according to claim 6, characterized in that, The text description of the keyword includes a first text description and a second text description. Generating the text description of the keyword based on the candidate detection category includes: The keywords are expanded using synonyms to obtain synonym keywords; Text generation processing is performed on the keywords and the synonyms to obtain a first text description including the keywords and a second text description including the synonyms.

8. The method according to claim 6, characterized in that, The text feature extraction process for the text description of the keywords, to obtain the text features of the text description of the keywords, includes: The text description is segmented to obtain the vocabulary units included in the text description; The vocabulary unit is subjected to feature embedding processing to obtain the unit embedding feature of the vocabulary unit; The unit embedding features of multiple lexical units are concatenated to obtain the text features of the keyword.

9. The method according to claim 6, characterized in that, The text features of the text description based on the keywords, used to determine the category features of the candidate detection category, include: If the keyword has multiple text descriptions, then the text features of the multiple text descriptions are fused to obtain the keyword features of the keyword; If the keyword has a text description, then the text features of the text description are determined as the keyword features of the keyword; If the keyword has synonyms, then the keyword features of the keyword and the keyword features of the synonyms are fused to obtain the category features of the candidate detection category; If the keyword does not have a synonym keyword, then the keyword feature of the keyword is determined as the category feature of the candidate detection category.

10. The method according to claim 3, characterized in that, The description parameters include the detection box confidence score. Determining the target detection result of the target image based on the target detection stitching result includes: The target image is subjected to image segmentation processing based on target image blocks to obtain at least one mask belonging to the target image block and the mask confidence of each mask, wherein the target image block is any image block from the detection box stitching result; A bounding box corresponding to the first mask is generated in the target image as a target detection box, wherein the first mask is any one of the at least one masks; Based on the detection box confidence and the mask confidence of the first mask included in the parameter splicing result, the target confidence of the target detection box is determined; The target detection result is generated based on the target detection bounding box and the target confidence score.

11. The method according to claim 10, characterized in that, The determination of the target confidence score of the target detection box based on the detection box confidence score and the mask confidence score of the first mask, including the parameter concatenation result, includes: The confidence scores of the detection boxes and the mask confidence scores of the first mask are standardized to obtain standardized detection box confidence scores and standardized mask confidence scores. The target confidence is obtained by multiplying the confidence of the standardized detection box with the corresponding relationship with the confidence of the standardized mask, wherein the corresponding relationship indicates that the first mask corresponding to the confidence of the standardized mask is within the detection box corresponding to the confidence of the standardized detection box.

12. A target detection device, characterized in that, The device includes: The closed set detection module is used to perform closed set target detection processing on the target image through the closed set target detection model to obtain the first target detection result; An open set detection module is used to perform open set target detection processing on the target image using an open set target detection model to obtain a second target detection result, wherein the detection accuracy of the first target detection result is higher than the detection accuracy of the second target detection result; The fusion processing module is used to fuse the first target detection result and the second target detection result to obtain the target detection result of the target image.

13. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the target detection method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by a processor, they implement the target detection method according to any one of claims 1 to 11.

15. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by a processor, they implement the target detection method according to any one of claims 1 to 11.

Images