A target detection method, apparatus, device, storage medium, and program product.

Abstract

This application provides a target detection method, apparatus, device, storage medium, and program product. Embodiments of this application can be applied to target detection scenarios in cloud technology, artificial intelligence, smart transportation, and vehicle-mounted systems. The method includes: extracting features from an image to be detected; extracting at least one supporting image feature from at least one supporting image of a reference target; enhancing each supporting image feature with shared information to obtain enhanced image features corresponding to each supporting image; determining at least one candidate image region from the image to be detected based on the features of the image to be detected; and determining the target image region where the reference target is located from the at least one candidate image region based on the enhanced image features of each supporting image and the candidate image features corresponding to each candidate image region. This application improves the accuracy of target detection.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a target detection method, apparatus, device, storage medium, and program product. Background Technology

[0002] Object detection is an important application of artificial intelligence, with applications in various fields such as autonomous driving and robotics. The amount of available labeled training data varies depending on the object category. For some object categories, there may be very little labeled training data. To address this, related technologies can determine the class prototype for each category using support data, and then achieve object detection by aggregating the class prototype with the features of the input data.

[0003] However, the supporting data may contain noise, which can have side effects on feature representation. For example, samples from different categories may be close to each other, or samples from the same category may differ in shape and angle. This can weaken the commonalities between different samples of the same category, ultimately affecting the accuracy of object detection. Summary of the Invention

[0004] This application provides a target detection method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy of target detection.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a target detection method, including:

[0007] From the image to be detected, features of the image to be detected are extracted, and from at least one support image of the reference target, at least one support image feature is extracted respectively; the support image is an image labeled with the location information of the reference target.

[0008] Enhancement of each supporting image feature is performed using shared information to obtain enhanced image features corresponding to each supporting image; wherein, the shared information characterizes the commonality of the benchmark target in at least one supporting image;

[0009] Based on the features of the image to be detected, at least one candidate image region is determined from the image to be detected;

[0010] Based on the enhanced image features of each of the supporting images and the candidate image features corresponding to each of the candidate image regions, the target image region where the reference target is located is determined from at least one of the candidate image regions.

[0011] This application provides a target detection device, including:

[0012] The feature extraction module is used to extract features of the image to be detected from the image to be detected, and to extract features of at least one support image from at least one support image of the reference target; the support image is an image labeled with the location information of the reference target.

[0013] The feature enhancement module is used to enhance the shared information for each of the supporting image features to obtain the enhanced image features corresponding to each supporting image; wherein, the shared information represents the commonality of the reference target in at least one of the supporting images;

[0014] A candidate generation module is used to determine at least one candidate image region from the image to be detected based on the features of the image to be detected;

[0015] The target determination module is used to determine the target image region where the reference target is located from at least one of the candidate image regions based on the enhanced image features of each of the supporting images and the candidate image features corresponding to each of the candidate image regions.

[0016] In some embodiments of this application, the feature enhancement module is further configured to perform feature concatenation on at least one of the supporting image features to obtain concatenated supporting features; perform feature extraction multiple times on the concatenated supporting features to obtain multiple mapped feature vectors; the mapped feature vectors contain feature components corresponding to each of the supporting image features; and enhance the shared information in each of the supporting image features based on the multiple mapped feature vectors to obtain the enhanced image features corresponding to each of the supporting images.

[0017] In some embodiments of this application, the target determination module is further configured to determine a first attention weight of the candidate image features corresponding to each candidate image region based on the enhanced image features of each supporting image; the first attention weight describes the similarity between the image content contained in the candidate image region and the benchmark target; the candidate image features are weighted using the first attention weight to obtain weighted image features of each candidate image region; and the target image region where the benchmark target is located is determined from at least one candidate image region based on the weighted image features of each candidate image region.

[0018] In some embodiments of this application, the target determination module is further configured to determine the image category and confidence level of each candidate image region based on the weighted image features of each candidate image region; and to determine at least one candidate image region that meets the target conditions as the target image region where the benchmark target is located; wherein the target conditions are that the image category matches the benchmark target and the confidence level is greater than the confidence level threshold.

[0019] In some embodiments of this application, the target determination module is further configured to concatenate the enhanced image features of each supporting image to obtain concatenated enhanced features, and concatenate the candidate image features of each candidate image region to obtain concatenated candidate features; perform at least two feature extractions on the concatenated enhanced features to obtain at least two enhanced feature vectors, and perform feature extraction on the concatenated candidate features to obtain candidate feature vectors; the candidate feature vectors include feature components corresponding to the candidate image features of each candidate image region; by applying a multi-head attention mechanism to the at least two enhanced feature vectors and the candidate feature vectors, the attention weight of each feature component of the candidate feature vectors is obtained; and the attention weight of each feature component of the candidate feature vectors is determined as the first attention weight of the candidate image feature corresponding to the candidate image region.

[0020] In some embodiments of this application, the target determination module is further configured to perform average processing on the enhanced image features of at least one of the supporting images to obtain average image features; calculate the feature similarity between the average image features and the candidate image features corresponding to each candidate image region; and determine the target image region where the reference target is located from at least one of the candidate image regions based on the feature similarity.

[0021] In some embodiments of this application, the candidate generation module is further configured to determine a second attention weight for each sub-image feature of the image to be detected based on the enhanced image features corresponding to each of the supporting images; to weight each sub-image feature using the second attention weight to obtain a weighted sub-image feature; and to determine at least one candidate image region from the image to be detected based on each weighted sub-image feature.

[0022] In some embodiments of this application, the feature extraction module is further configured to extract features from the labeled position of the reference target in at least one of the supporting images to obtain at least one target position feature; extract features from the image content of at least one of the supporting images to obtain at least one initial image feature; and fuse the at least one target position feature with the at least one initial image feature to obtain at least one supporting image feature.

[0023] In some embodiments of this application, the target detection device further includes: a feature storage module, used to store the enhanced image features corresponding to each of the supporting images; the enhanced image features are further used to determine the latest target image region where the reference target is located from at least one latest candidate image region of the latest image to be detected.

[0024] In some embodiments of this application, the baseline target includes: a virtual object; the target detection device further includes: a control generation module, used to generate control operations for the virtual object; the control operations are used to control the virtual object to complete target actions in a virtual scene, the target actions including: moving, releasing skills, and interacting with virtual props.

[0025] In some embodiments of this application, from extracting the features of the image to be detected from the image to be detected, to determining the target image region where the reference target is located from at least one of the candidate image regions based on the enhanced image features of each of the supporting images and the candidate image features corresponding to each of the candidate image regions, all are implemented by a target detection model;

[0026] The target detection device further includes: a model training module, used to acquire first training data and second training data; wherein the number of labeled data in the first training data is greater than the number of labeled data in the second training data; using the first training data to train an initial detection model to obtain an intermediate detection model; extracting candidate training data from the second training data; the candidate training data includes K labeled training instances, wherein the labeled training instances refer to images labeled with the location information of the reference target; using the target training data composed of the first training data and the candidate training data to train the intermediate detection model to obtain the target detection model.

[0027] This application provides a target detection device, including:

[0028] Memory, used to store executable instructions;

[0029] The processor, when executing executable instructions stored in the memory, implements the target detection method provided in the embodiments of this application.

[0030] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the target detection method provided in this application.

[0031] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the target detection method provided in this application.

[0032] The embodiments of this application have the following beneficial effects: the target detection device enhances the shared information of the support image features corresponding to each support image, so that the commonalities of different benchmark targets in different support images can be expressed in a more diverse and detailed way, that is, the ability to express the commonalities of different samples is improved. Thus, in subsequent processing, the enhanced image features can be used to more accurately determine whether there is a benchmark target in the candidate image region, and ultimately the accuracy of target detection is improved. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the architecture of the target detection system provided in the embodiments of this application;

[0034] Figure 2 This is provided by the embodiments of this application. Figure 1 A schematic diagram of the server structure in the diagram;

[0035] Figure 3 This is a flowchart illustrating the target detection method provided in the embodiments of this application. Figure 1 ;

[0036] Figure 4 This is a flowchart illustrating the target detection method provided in the embodiments of this application. Figure 2 ;

[0037] Figure 5 This is a flowchart illustrating the target detection method provided in the embodiments of this application. Figure 3 ;

[0038] Figure 6 This is a flowchart illustrating the target detection method provided in the embodiments of this application. Figure 4 ;

[0039] Figure 7 This is a schematic diagram illustrating the process of automating game testing provided in an embodiment of this application;

[0040] Figure 8 This is a schematic diagram illustrating the effect of detecting game characters provided in an embodiment of this application;

[0041] Figure 9This is a schematic diagram of the target detection model framework provided in the embodiments of this application;

[0042] Figure 10 This is a schematic diagram of the network structure for spatial query aggregation provided in an embodiment of this application;

[0043] Figure 11 This is a schematic diagram of the network structure for ROI query aggregation provided in the embodiments of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0048] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0049] 1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of responding with human-like intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0050] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0051] This application involves using artificial intelligence technology for target detection.

[0052] 2) Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes images to make them more suitable for human observation or instrument detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image detection, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0053] 3) Object detection is a technique for identifying objects in an image and their locations. The objects that object detection can detect depend on what objects the current task requires to detect. In this embodiment, object detection may be used to identify virtual objects in an image and their locations.

[0054] 4) Region of Interest (ROI) refers to the area in the image to be processed, delineated using shapes such as rectangles, circles, ellipses, or irregular polygons. The ROI is a key focus in object detection processing.

[0055] 5) The Regional Proposal Network (RPN) is used to provide ROIs for the image. That is, the RPN outputs a series of proposal boxes for the image and determines the proposal boxes as ROIs so that only ROIs are processed during object detection.

[0056] 6) A virtual scene is a virtual scene displayed (or provided) by an application when it runs on a terminal. This virtual scene can be a simulation of the real world, a semi-simulated virtual environment, or a purely virtual environment. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application embodiment does not limit the dimension of the virtual scene. A virtual scene can include sky, land, ocean, and virtual objects. The land can include environmental elements such as deserts and cities. Users can move within the virtual scene by controlling virtual objects.

[0057] 7) Virtual objects: These are interactive representations of people and objects within a virtual scene, or movable objects within that scene. These movable objects can be virtual characters, animals, anime characters, etc., such as people, animals, plants, oil drums, walls, and stones displayed in a virtual scene. A virtual object can be a virtual avatar representing the user within the virtual scene. A virtual scene can include multiple virtual objects, each with its own shape and volume, occupying a portion of the virtual scene's space. Virtual objects can be characters controlled through client-side operations; for example, a virtual character engaging in adversarial interactions within the virtual scene.

[0058] Object detection is an important application area of ​​artificial intelligence, with applications in various fields such as autonomous driving and robotics. Object detection is based on machine learning models; for example, deep learning models are trained using labeled sample data, and in practical applications, the trained models are used to detect different categories of objects in images, along with the objects' locations.

[0059] However, the amount of labeled training data available varies for different object categories. For some object categories, there may be very little labeled training data, resulting in poor performance of deep learning models and consequently lower object detection accuracy.

[0060] To address this issue, related technologies have proposed methods for object detection based on meta-learning. For example, a deep learning model is trained using a small amount of labeled support data. During inference, features are extracted from both the support data and the input data to be detected. Based on the averaging of features from different samples of each category, a class prototype for each category is obtained. Then, by aggregating the class prototype with the features of the input data, objects of a certain category and their locations are detected from the input data.

[0061] However, the supporting data may contain noise, which can have side effects on feature representation. For example, samples from different categories may be close to each other, or samples from the same category may differ in shape and angle. This can lead to poor representation of the commonalities between different samples of the same category when using feature representation, weakening the commonalities between different samples of the same category and ultimately affecting the accuracy of object detection.

[0062] Furthermore, when the features of different samples within the same category differ significantly, the class prototype obtained by averaging the features of different samples in each category will differ considerably from the true centers of the features of different samples, which will also affect the accuracy of object detection. Additionally, averaging the features of different samples in each category to obtain the class prototype for each category will result in poor diversity of class prototypes for each category, which will also affect the accuracy of object detection.

[0063] This application provides a target detection method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy of target detection. The following describes exemplary applications of the target detection device provided in this application. This target detection device can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server. The following describes exemplary applications when the target detection device is implemented as a server.

[0064] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the target detection system provided in this application embodiment. To support a target detection application, in the target detection system 100, terminals (terminals 400-1 and 400-2 are shown as examples) connect to the server 200 via a network 300. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. The target detection system 100 also includes a database 500 to provide data support to the server 200. The database 500 can be independent of the server 200 or configured within the server 200. Figure 1 This illustrates a scenario where the database is independent of server 200.

[0065] Terminals 400-1 and 400-2 are used to capture the image to be detected from the content displayed in the graphical interfaces 410-1 and 410-2 respectively, and send the image to be detected to the server 200 through the network 300.

[0066] Server 200 is used to extract features from the image to be detected and to extract at least one support image feature from at least one support image of the reference target, wherein the support image is an image labeled with the location information of the reference target; to enhance the shared information of each support image feature to obtain the enhanced image feature corresponding to each support image, wherein the shared information represents the commonality of the reference target in at least one support image; to determine at least one candidate image region from the image to be detected based on the features of the image to be detected; and to determine the target image region where the reference target is located from the at least one candidate image region based on the enhanced image feature of each support image and the candidate image feature corresponding to each candidate image region, so as to complete the target detection.

[0067] After obtaining the target image region, server 200 can also send the target image region to terminals 400-1 and 400-2, and terminals 400-1 and 400-2 can overlay the target image region on the image to be detected.

[0068] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals 400-1 and 400-2 may be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart home appliances, vehicle terminals, etc., but are not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0069] See Figure 2 , Figure 2 This is provided by the embodiments of this application. Figure 1 A schematic diagram of the structure of the server (an implementation of the target detection device) in the diagram. Figure 2 The server 200 shown includes at least one processor 210, memory 250, at least one network interface 220, and a user interface 230. The various components in server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0070] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0071] User interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0072] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0073] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0074] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0075] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0076] The network communication module 252 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 including: Bluetooth, Wi-Fi, and Universal Serial Bus (USB), etc.

[0077] Presentation module 253 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with user interface 230;

[0078] The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0079] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A target detection device 255 stored in memory 250 is shown. This device can be software in the form of programs or plugins, and includes the following software modules: feature extraction module 2551, feature enhancement module 2552, candidate generation module 2553, target determination module 2554, feature storage module 2555, control generation module 2556, and model training module 2557. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0080] In other embodiments, the target detection device provided in this application can be implemented in hardware. As an example, the target detection device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the target detection method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0081] In some embodiments, a server (an implementation of the target detection device) can implement the target detection method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as an object recognition APP; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0082] The embodiments of this application can be applied to target detection scenarios such as cloud technology, artificial intelligence, smart transportation, and vehicle-mounted systems. Below, the target detection method provided by the embodiments of this application will be described in conjunction with exemplary applications and implementations of the target detection equipment provided in the embodiments of this application.

[0083] See Figure 3 , Figure 3 This is a flowchart illustrating the target detection method provided in the embodiments of this application. Figure 1 , will combine Figure 3 The steps shown are explained.

[0084] S101. Extract the features of the image to be detected from the image to be detected, and extract at least one support image feature from at least one support image of the reference target.

[0085] This application embodiment is implemented in the scenario of object detection in an image, such as detecting whether a virtual object or a certain object exists in an image. The object detection device first extracts features from the image to be detected to obtain the features of the image to be detected. At the same time, the object detection device acquires at least one supporting image of the reference target and extracts features from each supporting image to obtain at least one supporting image feature corresponding to at least one supporting image.

[0086] It should be noted that the baseline target is the target object that the target detection device needs to detect. The baseline target can be an everyday object, such as a television or computer, or a virtual object in a virtual scene, such as a game character or game prop.

[0087] The image to be detected can be captured in real time by the target detection device through a connected image acquisition device. For example, a vehicle-mounted camera can capture images of the road to determine whether there are obstacles in the road through a target detection process. The image to be detected can also be received by the target detection device from other electronic devices. For example, a terminal can take a screenshot of a virtual scene in a graphical interface and send the screenshot to the target detection device as the image to be detected.

[0088] In some embodiments, the object detection device can extract features from the image to be detected using a convolutional neural network (CNN) to obtain the features of the image to be detected. In other embodiments, the object detection device can also extract color histograms or texture features from the features of the image to be detected, and use the extracted features as the features of the image to be detected.

[0089] It should be noted that the support images are images with the location information of the baseline target marked. In the support images, the location of the baseline target has already been marked using bounding boxes. The angle and shape of the baseline target can be different in different support images. For example, for the same game character, it can have different poses in different support images, or for the same fruit, it can have different colors in different support images.

[0090] The process by which the target detection device extracts supporting image features from supporting images is similar to the process of extracting features from the image to be detected, and will not be repeated here.

[0091] It is understood that the features of the image to be detected can be feature maps or feature vectors. Correspondingly, the supporting image features can be feature maps or feature vectors. This application does not limit the specific features.

[0092] S102. Enhance the shared information for each supporting image feature to obtain the enhanced image feature corresponding to each supporting image.

[0093] The target detection device first identifies shared information among at least one support image, and then enhances the shared information contained in the features of each support image to make the commonalities of the baseline target in different support images more diverse and detailed, thereby strengthening the influence of the shared information in each support image. The support image features that have been enhanced with the shared information are then used as enhanced image features.

[0094] In other words, by enhancing the shared information in the supporting image features, the target detection device can enrich the expressive power of the shared information between different supporting images. As a result, the enhanced image features can express the commonalities of the benchmark target among different supporting images in a more detailed way compared to the original supporting image features.

[0095] It is understandable that shared information represents the commonalities among at least one supporting image of the baseline target, that is, the characteristics shared by different supporting images. Therefore, shared information is an important condition for determining whether an image region contains the baseline target. For example, the same virtual object may have different poses, actions, etc. in different supporting images, but the appearance of the virtual object should be similar in different supporting images. Thus, appearance is the commonality among different supporting images, and the features corresponding to the appearance are the shared information.

[0096] In some embodiments, the object detection device can determine the representation of shared information from different perspectives by mapping each supporting image feature to a different feature space and processing the mapping vectors of the different feature spaces through an attention mechanism.

[0097] In other embodiments, the target detection device may first extract shared information from the supporting image, and then process the shared information using different feature processing models, such as pooling the shared information using a pooling model, convolutional processing the shared information using a convolutional model, etc., and then add all the processed features to the tail of the supporting image features to obtain enhanced image features.

[0098] S103. Based on the features of the image to be detected, determine at least one candidate image region from the image to be detected.

[0099] The target detection device analyzes and processes the features of the image to be detected to determine which image regions in the image may contain the reference target, and identifies these image regions as candidate image regions. In other words, the target detection device first preliminarily identifies some regions of interest, i.e., candidate image regions, from the image to be detected based on the features. These candidate image regions need to be further combined with supporting image features to determine whether the reference target exists.

[0100] In some embodiments, the object detection device can input the features of the image to be detected into a trained Region Proposal Network (RPN) to determine at least one candidate image region. In other embodiments, the object detection device can analyze each sub-feature of the features of the image to be detected and determine the image regions corresponding to the sub-features whose differences from other sub-features are greater than a threshold as candidate image regions (because image regions in the image to be detected that are significantly different from other regions are very likely to be regions of interest that need to be detected in detail).

[0101] S104. Based on the enhanced image features of each supporting image and the candidate image features corresponding to each candidate image region, determine the target image region where the reference target is located from at least one candidate image region.

[0102] After obtaining the enhanced image features of each supporting image, the target detection device uses these enhanced image features, along with the candidate image features corresponding to each candidate image region, to determine whether a baseline target appears in each candidate image region. Candidate image regions where the baseline target appears are then identified as target image regions. In this way, the target detection device completes the target detection process for the baseline target.

[0103] It is understandable that, compared to related technologies, the features of different samples of the same category differ significantly, resulting in a weaker influence of the commonalities among different samples, which ultimately affects the accuracy of target detection. In this embodiment, the target detection device enhances the shared information of the support image features corresponding to each support image, so that the commonalities of different benchmark targets in different support images can be expressed in a more diverse and detailed manner. That is, it improves the ability to express the commonalities of different samples, thereby enabling a more detailed determination of whether a benchmark target exists in the candidate image region by means of enhanced image features during subsequent processing, ultimately improving the accuracy of target detection.

[0104] based on Figure 3 See Figure 4 , Figure 4 This is a flowchart illustrating the target detection method provided in the embodiments of this application. Figure 2 In some embodiments of this application, the shared information is enhanced for each supporting image feature to obtain the enhanced image feature corresponding to each supporting image. The specific implementation process of S102 may include: S1021-S1023, as follows:

[0105] S1021. At least one supporting image feature is stitched together to obtain stitched supporting features.

[0106] The object detection device can select any one of the three tensor dimensions (e.g., width, height, number of channels) that support image features, and then stitch the supporting image features on the selected tensor dimension. After the stitching is completed, the stitched supporting features are obtained.

[0107] S1022. Perform multiple feature extractions on the concatenated support features to obtain multiple mapped feature vectors.

[0108] Understandably, object detection devices can utilize multiple different network models to extract features from the concatenated support features, resulting in multiple mapped feature vectors. These multiple network models can have the same network structure but different model parameters; for example, convolutional layers with the same structure but different model parameters. Alternatively, multiple network models can have both different network structures and different model parameters.

[0109] It should be noted that the mapped feature vector contains feature components corresponding to each supporting image feature, that is, the feature components in the mapped feature vector correspond one-to-one with the supporting image features.

[0110] S1023. Based on multiple mapping feature vectors, the shared information in each support image feature is enhanced to obtain the enhanced image feature corresponding to each support image.

[0111] Object detection devices can input multiple mapped feature vectors into a multi-head attention model. This model maps these vectors to different feature spaces for interaction, allowing for a more diverse representation of shared information across these spaces, resulting in enhanced image features. Furthermore, for each mapped feature vector, the device can query the feature components corresponding to shared information from the remaining mapped feature vectors, select one of these components, and fuse it into each mapped feature vector. The fused mapped feature vectors are then restored to individual image features, yielding enhanced image features.

[0112] based on Figure 3 See Figure 5 , Figure 5 This is a flowchart illustrating the target detection method provided in the embodiments of this application. Figure 3 In some embodiments of this application, the process of determining the target image region where the reference target is located from at least one candidate image region based on the enhanced image features of each supporting image and the candidate image features corresponding to each candidate image region, i.e., the implementation process of S104, may include: S1041-S1043, as follows:

[0113] S1041. Based on the enhanced image features of each supporting image, determine the first attention weight of the candidate image features corresponding to each candidate image region.

[0114] Object detection devices can calculate enhanced image features supporting the image, cosine similarity between candidate image features corresponding to each candidate image region, and assign corresponding first attention weights to the candidate image features based on the similarity. Object detection devices can also calculate feature distances between enhanced image features and candidate image features corresponding to each candidate image region, and assign corresponding first attention weights to the candidate image features based on the feature distances. However, both feature distance and cosine similarity indicate the similarity between the image content in the candidate image region and the baseline target. Therefore, the first attention weight describes the degree of similarity between the image content in the candidate image region and the baseline target.

[0115] S1042. The candidate image features are weighted using the first attention weight to obtain the weighted image features of each candidate image region.

[0116] S1043. Based on the weighted image features of each candidate image region, determine the target image region where the reference target is located from at least one candidate image region.

[0117] The target detection device uses a first attention weight to weight the features of the candidate image, so as to emphasize the similarity between the image content contained in the candidate image region and the reference target, so as to facilitate the subsequent determination of the target image region where the reference target is located.

[0118] In some embodiments of this application, the specific implementation process of determining the target image region where the reference target is located from at least one candidate image region based on the weighted image features of each candidate image region, i.e., S1043, may include: S1043a-S1043b (not shown in the figure), as follows:

[0119] S1043a. Based on the weighted image features of each candidate image region, determine the image category and confidence level of each candidate image region.

[0120] The object detection device identifies the category of image content contained in candidate image regions based on weighted image features, and assigns the identified category to each candidate image region. Simultaneously, the device outputs an image category confidence score based on the weighted image features, indicating the degree of credibility of the image category. This confidence score can be understood as being based on the similarity between the image content contained in the candidate image region and the baseline target.

[0121] S1043b: Determine at least one candidate image region that meets the target conditions as the target image region where the reference target is located.

[0122] The target condition is that the image category matches the baseline target, and the confidence level is greater than the confidence threshold. In other words, the target detection device determines whether the image category corresponding to each candidate image region matches the baseline target, and whether the confidence level corresponding to the image category is greater than the confidence threshold. Only when both conditions are met will the target detection device identify the candidate image region as the target image region.

[0123] It is understood that the target detection device can determine that the image category matches the reference target when the image category is the same as the reference target category, or it can determine that the image category matches the reference target category when the image category contains the reference target category. Furthermore, the confidence threshold can be set according to actual conditions, and this application embodiment does not impose any limitations on it.

[0124] In some embodiments of this application, the first attention weight of the candidate image features corresponding to each candidate image region is determined based on the enhanced image features of each supporting image. The specific implementation process of S1041 may include: S1041a-S1041d (not shown in the figure), as follows:

[0125] S1041a. The enhanced image features of each supporting image are concatenated to obtain concatenated enhanced features, and the candidate image features of each candidate image region are concatenated to obtain concatenated candidate features.

[0126] It should be noted that the object detection device needs to use the same stitching method to obtain stitched enhanced features and stitched candidate features. For example, when the object detection device stitches the enhanced image features of each supporting image end-to-end to obtain stitched enhanced features, it also needs to stitch the candidate image features of each candidate image region end-to-end to obtain stitched candidate features.

[0127] S1041b: Perform at least two feature extractions on the splicing enhanced features to obtain at least two enhanced feature vectors, and perform feature extraction on the splicing candidate features to obtain candidate feature vectors.

[0128] Target detection equipment can use at least two feature extraction models with different network structures and model parameters to extract features from spliced ​​and enhanced features, or it can use at least two feature extraction models with the same network structure but different model parameters to extract features from spliced ​​and enhanced features.

[0129] It is understandable that the candidate feature vector includes feature components corresponding to the candidate image features of each candidate image region.

[0130] S1041c: By applying a multi-head attention mechanism to at least two enhanced feature vectors and candidate feature vectors, the attention weight of each feature component of the candidate feature vector is obtained.

[0131] The multi-head attention mechanism is implemented through a multi-head attention model, which contains multiple parallel attention modules. The object detection device uses at least two enhanced feature vectors and candidate feature vectors as inputs to each attention module in the multi-head attention model. Each attention module outputs a sub-attention weight corresponding to that attention module for each feature component of the candidate feature vector, and the sub-attention weights corresponding to each attention module are fused to obtain the attention weight corresponding to each feature component.

[0132] S1041d: Determine the attention weight of each feature component of the candidate feature vector as the first attention weight of the candidate image feature corresponding to the candidate image region.

[0133] Since the feature components in the candidate feature vector correspond one-to-one with the candidate image features, the target detection device can directly determine the attention weight of each feature component as the first attention weight of the corresponding candidate image feature, so as to facilitate the subsequent weighting of each candidate image feature to obtain the weighted image feature.

[0134] Understandably, compared to related technologies that generate only the same class prototype for different samples of a category, in this embodiment, the target detection device uses the enhanced image features of each supporting image as a class prototype, and determines a first attention weight for the candidate image features using the class prototype. The first attention weight emphasizes the similarity between the content contained in the candidate image region and the baseline target, thereby achieving the aggregation of candidate image features and supporting image features, and thus determining the target image region where the baseline target is located. In this way, the class prototypes will be more diverse, that is, the diversity of class prototypes in the same category will not be lost, which helps to improve the accuracy of target detection.

[0135] In some embodiments of this application, the specific implementation process of determining the target image region where the reference target is located from at least one candidate image region based on the enhanced image features of each supporting image and the candidate image features corresponding to each candidate image region, i.e., S104, may include: S1044-S1046 (not shown in the figure), as follows:

[0136] S1044. Average the enhanced image features of at least one supporting image to obtain average image features.

[0137] The target detection device assigns the same weight to the enhanced image features of each supporting image (the sum of all weights is 1), and then performs a weighted summation of all enhanced image features using this weight to obtain the average image features.

[0138] S1045. Calculate the feature similarity between the average image features and the candidate image features corresponding to each candidate image region.

[0139] Next, the object detection device calculates the similarity between the average image features and the candidate image features corresponding to each candidate image region. The calculated similarity is the feature similarity. It can be understood that the object detection device can obtain the feature similarity by calculating cosine similarity or feature distance.

[0140] S1046. Based on feature similarity, determine the target image region where the reference target is located from at least one candidate image region.

[0141] The target detection device can determine the candidate image region corresponding to the highest feature similarity as the target image region where the reference target is located, or it can determine the candidate image region whose corresponding feature similarity exceeds the similarity threshold as the target image region. This application embodiment does not limit this.

[0142] It is understood that in the embodiments of this application, the target detection device performs average processing on the enhanced image features of the shared information, which have more diverse and detailed features. The resulting average image features, i.e., class prototypes, are more accurate, thereby maintaining the accuracy of target detection at a high level.

[0143] based on Figure 3 See Figure 6 , Figure 6 This is a flowchart illustrating the target detection method provided in the embodiments of this application. Figure 4 In some embodiments of this application, after enhancing the shared information for each supporting image feature to obtain the enhanced image features corresponding to each supporting image, and before determining the target image region where the reference target is located from at least one candidate image region based on the enhanced image features of each supporting image and the candidate image features corresponding to each candidate image region, i.e., after S102 and before S104, the method may further include: S105-S107, as follows.

[0144] S105. Based on the enhanced image features corresponding to each supporting image, determine the second attention weight for each sub-image feature of the image to be detected.

[0145] The target detection device concatenates the enhanced image features corresponding to each supporting image into a concatenated enhanced feature. At the same time, it extracts multiple sub-image features from the features of the image to be detected. A multi-head attention mechanism is applied to the concatenation result of the multiple sub-image features and the concatenated enhanced feature to obtain the second attention weight corresponding to each sub-image feature.

[0146] It should be noted that the process of applying a multi-head attention mechanism to the stitching results of multiple sub-image features and the stitching enhancement features to obtain the second attention weight is quite similar to the process of applying a multi-head attention mechanism to at least two enhanced feature vectors and candidate feature vectors to obtain the first attention weight in the above steps, and will not be repeated here.

[0147] S106. Using the second attention weight, weight each sub-image feature is weighted to obtain weighted sub-image features.

[0148] S107. Based on the features of each weighted sub-image, determine at least one candidate image region from the image to be detected.

[0149] The target detection device uses a second attention weight to weight each sub-image feature, thereby increasing the significance of sub-image features that are similar to the enhanced image features in the target image. This identifies which image region in the target image is most similar to the region containing the reference target. Thus, when determining at least one candidate image region from the target image based on the weighted sub-image features, regions similar to the supporting image are prioritized, improving the accuracy of candidate image regions and ultimately enhancing the accuracy of target detection.

[0150] In some embodiments of this application, the specific implementation process of extracting at least one support image feature from at least one support image of the reference target, i.e., S101, may include: S1011-S1013 (not shown in the figure), as follows:

[0151] S1011. Extract features from the labeled position of the reference target in at least one supporting image to obtain at least one target position feature.

[0152] S1012. Extract features from the image content of at least one supporting image to obtain at least one initial image feature.

[0153] Since the support image contains the location of the baseline target, that is, the support image contains both the information of the labeled location of the baseline target and the information of the image content of the baseline target, the target detection device will perform feature extraction on each support image simultaneously in the dimensions of labeled location and image content, so as to obtain the target location features and initial image features of each support image respectively.

[0154] S1013. At least one target location feature is fused with at least one initial image feature to obtain at least one supporting image feature.

[0155] The target detection device fuses the target location features of each supporting image with the initial image features. For example, it can stitch the target location features and the initial image features together, or it can weight the target location features and the initial image features to obtain supporting image features that simultaneously contain information about the appearance of the reference target and the location of the reference target. This allows for the subsequent determination of the region of interest, i.e., the determination of the candidate image region, based on the supporting image features.

[0156] In some embodiments of this application, after enhancing the shared information for each supporting image feature to obtain the enhanced image features corresponding to each supporting image, i.e. after S102, the method may further include: S108 (not shown in the figure), as follows:

[0157] S108. Store the enhanced image features corresponding to each supported image.

[0158] It should be noted that, in this embodiment, the enhanced image features are also used to determine the latest target image region where the reference target is located from at least one of the latest candidate image regions of the latest image to be detected. That is, in addition to determining the latest target image region where the reference target is located for the current image to be detected, the enhanced image features can be reused, i.e., to help determine the latest target image region where the reference target is located from at least one of the latest candidate image regions of the latest image to be detected input for the next round of target detection. Thus, when performing target detection on the latest image to be detected, the process of extracting supporting image features and the enhancement process of sharing information can be skipped, and the enhanced image features can be obtained directly, saving target detection time.

[0159] In some embodiments of this application, the benchmark target includes a virtual object. In this case, after determining the target image region where the benchmark target is located from at least one candidate image region based on the enhanced image features of each supporting image and the candidate image features corresponding to each candidate image region, i.e., after S104, the method may further include: S109, as follows:

[0160] S109, Control operations are generated for virtual objects.

[0161] The control operation is used to control virtual objects to complete target actions in a virtual scene. These target actions include movement, skill activation, and interaction with virtual props. In other words, when the target detection device detects a virtual object in the image to be inspected, it can automatically control the virtual object to enable it to perform various behaviors within the virtual scene. This step can be applied to automated testing of applications that provide virtual scenes to accelerate the testing process.

[0162] In some embodiments of this application, from extracting the features of the image to be detected from the image to be detected, up to determining the target image region where the reference target is located from at least one candidate image region based on the enhanced image features of each supporting image and the candidate image features corresponding to each candidate image region, all are implemented through a target detection model. That is, the entire target detection process is implemented through a target detection model. At this time, before extracting the features of the image to be detected from the image to be detected, i.e. before S101, the method may further include: S201-S204, as follows:

[0163] S201. Obtain the first training data and the second training data.

[0164] It should be noted that the amount of labeled data in the first training dataset is greater than the amount of labeled data in the second training dataset. That is, the first training dataset contains a sufficient number of images with labeled data, while the second training dataset contains only a small number of labeled images. The labeled data is used to annotate the reference targets contained in the images.

[0165] S202. Using the first training data, train the initial detection model to obtain the intermediate detection model.

[0166] Understandably, the initial detection model can be a CNN model with its parameters initialized, meaning the model parameters in the CNN model have not been trained at all. Alternatively, the initial detection model can be a CNN model pre-trained using some unlabeled data, meaning the model parameters in the CNN model have been trained but have not yet reached their optimal performance for the object detection task.

[0167] S203. Extract candidate training data from the second training data.

[0168] The target detection device extracts K labeled training instances of the baseline target from the second training data, and uses these K labeled training instances to form candidate training data. In other words, the candidate training data includes K labeled training instances. Here, a labeled training instance refers to an image labeled with the location information of the baseline target.

[0169] S204. Using the target training data composed of the first training data and the candidate training data, train the intermediate detection model to obtain the target detection model.

[0170] Finally, the object detection device integrates the first training data and candidate training data into a new training data, namely the target training data. Then, it uses the intermediate detection model to perform forward prediction on the target training data, and backpropagates the differences between the prediction results and the labeled data in the target training data to adjust the model parameters of the intermediate detection model, completing one training iteration. This process is repeated until the training termination condition is met, thus obtaining the object detection model. The training termination condition can be reaching a preset number of iterations or the model accuracy reaching a preset accuracy; this embodiment does not limit the specific conditions.

[0171] Understandably, the structure of an object detection model can be configured according to actual needs. For example, the model could consist of two backbone networks, an aggregation network, and an RPN network. The two backbone networks extract features from the image to be detected and the support images, respectively. The RPN network determines candidate image regions, and the aggregation network determines enhancement image features. Based on the enhancement image features and candidate image features, the target image region is determined. Alternatively, the model could consist of K+1 backbone networks, two aggregation networks, and an RPN network. The K+1 backbone networks extract features from the image to be detected and K support images in parallel. One aggregation network determines enhancement image features, aggregates these features with the features of the image to be detected, and inputs them into the RPN network to obtain candidate image regions. The other aggregation network aggregates the candidate image features with the enhancement image features to determine the final target image region.

[0172] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0173] This application embodiment achieves automated testing by detecting game characters in a game application. See also... Figure 7 , Figure 7 This is a schematic diagram illustrating the process of automating game testing provided in this application embodiment. The game runs on terminal 7-1. Terminal 7-1 takes a screenshot of the game scene (virtual scene) and sends the screenshot image 7-2 (image to be tested) to server 7-3 (target detection device). The server identifies the screenshot image 7-2. When it identifies a game character (virtual object) in the game scene, it issues action commands 7-4 to terminal 7-1, such as casting skills, moving, or using items (control operations), to automatically run the game on terminal 7-1 and perform automated testing.

[0174] Figure 8 This is a schematic diagram illustrating the effect of detecting game characters according to an embodiment of this application. The server detects the game characters in screenshot image 8-2, including game characters 8-21 and 8-22, based on and supporting the game characters provided in image 8-1.

[0175] The detection process for game characters will be explained below.

[0176] The process of detecting game characters can be viewed as a small-sample object detection process. That is, considering two different datasets Db and Dn, where Dd has abundant labeled data Cb, and Dn is a new dataset with very little labeled data Cn. Here, we assume that there is no overlap between Cb and Cn, i.e. Therefore, few-sample object detection aims to train Dn using the knowledge of Db to identify and locate instances of the new class Cn, thus achieving object detection. In other words, Dn is the query image, i.e. Figure 7 and Figure 8 The screenshot image in the scene, Db is the supporting image, and Db consists of K unlabeled instances of each category.

[0177] Figure 9 This is a schematic diagram of the target detection model framework provided in an embodiment of this application. See also... Figure 9The query image 9-1 is input into the backbone network 9-3, and the K support images 9-2 are input into the backbone network 9-4 for feature extraction. The feature map output by the backbone network 9-3 has a scale of (1, Hq, Wq, C). The backbone network 9-4 outputs K feature maps. For each of the K feature maps (at least one initial image feature), the features obtained by region pooling of the support images (at least one target location feature) is combined with the labeled data of the support images, i.e., the ground truth bounding boxes 9-5 in the support images, to obtain features of (K, Hroi, Wroi, C) (K support image features). In some cases, the feature map (1, Hq, Wq, C) (the features of the image to be detected) is directly input into the RPN network 9-6 without performing spatial query aggregation 9-7 with the feature map (K, Hroi, Wroi, C) (the supporting image features). In other cases, the feature maps (1, Hq, Wq, C) and (K, Hroi, Wroi, C) need to undergo spatial query aggregation 9-7 before being input into the RPN network 9-6. That is, spatial query aggregation 9-7 is optional. The RPN network (9-6) is used to provide candidate regions. After pooling operations (9-8), the images in the candidate regions are used to obtain a feature map (Nroi, Hroi, Wroi, C) (candidate image features). This feature map is then aggregated with the feature map (K, Hroi, Wroi, C) for ROI query (9-9). The aggregation result is then processed for ROI (9-10) to obtain the bounding box category (image category) and confidence score, finally yielding the output result (9-11).

[0178] It should be noted that spatial query aggregation is used to spatially enhance the saliency of features corresponding to regions similar to supporting images within the query features, while ROI query aggregation further emphasizes the similarity between query features and supporting features. Both the spatial query aggregation network structure and the ROI query aggregation network structure consist of an internal attention module and a query support attention module.

[0179] The goal of the internal attention module is to refine the support vectors of each support image with the help of other support vectors. This is because the properties of the support images differ, causing some support feature vectors to differ too much from others, which can negatively impact object detection performance. Therefore, the server enhances the shared information of support vectors through a multi-head attention mechanism to improve model performance. The internal attention module is designed as a shallow transformation encoder, which consists of a multi-head attention network and a multilayer perceptron.

[0180] The query support attention module aggregates the features and supporting features of the query image through an attention mechanism. This aggregation process is accomplished by performing a dot product on each query feature vector using all the supporting features. In practice, the query support attention module can be designed as a shallow transformation decoder, consisting of a multilayer perceptron and a multi-head attention network, with layer normalization processing performed.

[0181] Figure 10 This is a schematic diagram of the network structure for spatial query aggregation provided in an embodiment of this application. The network structure 10-1 for spatial query aggregation includes an internal attention module 10-11 and a query support attention module 10-12. The input to the internal attention model 10-11 is the features of (K, Hroi, Wroi, C). Figure 10-11 1. The server pools each (Hroi, Wroi, C) feature map to obtain K (1, C) vectors of size 10-112 (supporting image features), and then concatenates them to form a (K, 1, C) feature map. Figure 10-11 3 (Feature Concatenation). Next, three different multilayer perceptrons, namely multilayer perceptron 10-114, multilayer perceptron 10-115, and multilayer perceptron 10-116, are used for feature extraction, yielding vectors K1, V1, and Q1 (multiple mapped feature vectors). Then, a multi-head attention network 10-117 is applied to these three mapped feature vectors to obtain new features. Figure 10-11 8. New features Figure 10-11 Each vector (enhanced image feature) contained in 8 is obtained by enhancing the shared information in vectors 10-112.

[0182] The query supports the attention module 10-12 with a subset of inputs of (1, H, W, C) as features. Figure 10-1 21, the server converts it into a feature of (1, HW, C). Figure 10-1 22. Vector Q2 is extracted using multilayer perceptron 10-123. New features are then processed using multilayer perceptrons 10-124 and 10-125. Figure 10-11 8 is processed to obtain vectors K2 and V2, and then processed again based on a multi-head attention network 10-126 to obtain weights (second attention weights). Based on the weights, the feature (1, HW, C) is obtained. Figure 10-1 27, where the significance of each vector (sub-image feature) changes compared to each vector in the original features 10-122 (i.e., weighted sub-image features are obtained), and then the feature map 10-127 is restored to a feature map of (1, HW, C). Figure 10-1 28, to determine the ROI region (obtain candidate image regions).

[0183] Figure 11This is a schematic diagram of the network structure for ROI query aggregation provided in an embodiment of this application. The network structure 11-1 includes... Figure 10 The internal attention modules 10-11 in the middle, and with Figure 10 The query support attention modules 10-12 are slightly different from the query support attention modules 11-12. The input to the query support attention modules 11-12 is features of (Nroi, Hroi, Wroi, C). Figure 11-1 21. The server performs pooling on Nroi feature maps and concatenates the pooled vectors 11-122 to form a feature map of (Nroi, 1, C). Figure 11-1 23 (concatenate candidate features), and use multilayer perceptron 11-124 to process the features. Figure 11-1 23 is processed to obtain vector Q3 (candidate feature vector). Multilayer perceptrons 11-125 and 11-126 are then used to process the features... Figure 10-11 8. After processing, vectors K3 and V3 (enhanced feature vectors) are obtained. These vectors are then processed by a multi-head attention network (11-127) to obtain weights (first attention weights). Based on these weights, the feature vector (Nroi, 1, C) is obtained. Figure 11-1 28, Regarding features Figure 11-1 The 28 is segmented, and each resulting vector 11-129 (weighted image features) is the input for ROI processing.

[0184] The training process of the object detection model will be explained below.

[0185] First, the network (initial detection network) is trained using sufficient labeled data Db (first training data), at which point the labeled data is Cb. Second, the network is fine-tuned using small sample data Db (second training data). At this stage, a balanced dataset (target training data) is formed from K instances (candidate training data) of each class in Db and Dn, thus determining the total number of classes during training, i.e., Ctrain = Cb∪Cn. The training strategy adopted for these two stages is as follows:

[0186] Each training batch consists of n or K instances and a query image, and different training processes are selected based on the baseline framework of the object detection model (optional FewX or FsDetView).

[0187] For example, the baseline FewX is trained using a 2-way, K-instance batch, where each batch consists of a triple (qc1, sc1, sc2). Here, c1 and c2 are different classes extracted from Ctrain, qc1 identifies the query data including instances of class c1, and sc1 and sc2 represent the 2-way, K-instance training data, i.e., |sc1| = |sc2| = K. For the baseline FsDetView, a batch includes one query image and all classes, i.e., one query image and all supporting images for each class.

[0188] During inference, all supporting images are passed to the backbone network and the internal attention model once. The feature vectors output by the internal attention module are stored as class prototypes for repeated use.

[0189] It is understood that in the embodiments of this application, data such as screenshots of user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0190] The following description continues to illustrate the exemplary structure of the target detection device 255 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the target detection device 255 in the memory 250 may include:

[0191] The feature extraction module 2551 is used to extract features of the image to be detected from the image to be detected, and to extract features of at least one support image from at least one support image of the reference target; the support image is an image labeled with the location information of the reference target.

[0192] The feature enhancement module 2552 is used to enhance the shared information for each of the supporting image features to obtain the enhanced image features corresponding to each of the supporting images; wherein, the shared information characterizes the commonality of the reference target in at least one of the supporting images;

[0193] The candidate generation module 2553 is used to determine at least one candidate image region from the image to be detected based on the features of the image to be detected;

[0194] The target determination module 2554 is used to determine the target image region where the reference target is located from at least one of the candidate image regions based on the enhanced image features of each of the supporting images and the candidate image features corresponding to each of the candidate image regions.

[0195] In some embodiments of this application, the feature enhancement module 2552 is further configured to perform feature concatenation on at least one of the supporting image features to obtain concatenated supporting features; perform feature extraction multiple times on the concatenated supporting features to obtain multiple mapped feature vectors; the mapped feature vectors contain feature components corresponding to each of the supporting image features; and enhance the shared information in each of the supporting image features based on the multiple mapped feature vectors to obtain the enhanced image features corresponding to each of the supporting images.

[0196] In some embodiments of this application, the target determination module 2554 is further configured to determine a first attention weight of the candidate image features corresponding to each candidate image region based on the enhanced image features of each supporting image; the first attention weight describes the similarity between the image content contained in the candidate image region and the benchmark target; the candidate image features are weighted using the first attention weight to obtain weighted image features of each candidate image region; and the target image region where the benchmark target is located is determined from at least one candidate image region based on the weighted image features of each candidate image region.

[0197] In some embodiments of this application, the target determination module 2554 is further configured to determine the image category and confidence level of each candidate image region based on the weighted image features of each candidate image region; and to determine at least one candidate image region that meets the target conditions as the target image region where the benchmark target is located; wherein the target conditions are that the image category matches the benchmark target and the confidence level is greater than the confidence level threshold.

[0198] In some embodiments of this application, the target determination module 2554 is further configured to concatenate the enhanced image features of each supporting image to obtain concatenated enhanced features, and concatenate the candidate image features of each candidate image region to obtain concatenated candidate features; perform at least two feature extractions on the concatenated enhanced features to obtain at least two enhanced feature vectors, and perform feature extraction on the concatenated candidate features to obtain candidate feature vectors; the candidate feature vectors include feature components corresponding to the candidate image features of each candidate image region; by applying a multi-head attention mechanism to at least two enhanced feature vectors and the candidate feature vectors, the attention weight of each feature component of the candidate feature vectors is obtained; and the attention weight of each feature component of the candidate feature vectors is determined as the first attention weight of the candidate image features corresponding to the candidate image region.

[0199] In some embodiments of this application, the target determination module 2554 is further configured to perform average processing on the enhanced image features of at least one of the supporting images to obtain average image features; calculate the feature similarity between the average image features and the candidate image features corresponding to each candidate image region; and determine the target image region where the reference target is located from at least one of the candidate image regions based on the feature similarity.

[0200] In some embodiments of this application, the candidate generation module 2553 is further configured to determine a second attention weight for each sub-image feature of the image to be detected based on the enhanced image features corresponding to each of the supporting images; to use the second attention weight to weight each sub-image feature to obtain a weighted sub-image feature; and to determine at least one candidate image region from the image to be detected based on each weighted sub-image feature.

[0201] In some embodiments of this application, the feature extraction module 2551 is further configured to extract features from the labeled position of the reference target in at least one of the supporting images to obtain at least one target position feature; extract features from the image content of at least one of the supporting images to obtain at least one initial image feature; and fuse the at least one target position feature with the at least one initial image feature to obtain at least one supporting image feature.

[0202] In some embodiments of this application, the target detection device 255 further includes: a feature storage module 2555, used to store the enhanced image features corresponding to each of the supporting images; the enhanced image features are further used to determine the latest target image region where the reference target is located from at least one latest candidate image region of the latest image to be detected.

[0203] In some embodiments of this application, the baseline target includes a virtual object; the target detection device 255 further includes a control generation module 2556, used to generate control operations for the virtual object; the control operations are used to control the virtual object to complete target actions in a virtual scene, the target actions include: moving, releasing skills, and interacting with virtual props.

[0204] In some embodiments of this application, from extracting the features of the image to be detected from the image to be detected, to determining the target image region where the reference target is located from at least one of the candidate image regions based on the enhanced image features of each of the supporting images and the candidate image features corresponding to each of the candidate image regions, all are implemented by a target detection model;

[0205] The target detection device 255 further includes: a model training module 2557, used to acquire first training data and second training data; wherein the number of labeled data in the first training data is greater than the number of labeled data in the second training data; using the first training data to train an initial detection model to obtain an intermediate detection model; extracting candidate training data from the second training data; the candidate training data includes K labeled training instances, wherein the labeled training instances refer to images labeled with the location information of the reference target; using the target training data composed of the first training data and the candidate training data to train the intermediate detection model to obtain the target detection model.

[0206] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the target detection method described in this application.

[0207] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the target detection method provided in this application. For example, ... Figure 3 The target detection method is shown.

[0208] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0209] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0210] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0211] As an example, executable instructions can be deployed to execute on a single electronic device (an implementation of a target detection device), or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0212] In summary, through the embodiments of this application, the target detection device enhances the shared information of the supporting image features corresponding to each supporting image, so that the commonalities of different benchmark targets in different supporting images can be expressed in a more diverse and detailed manner. That is, it improves the ability to express the commonalities of different samples, thereby enabling a more detailed determination of whether a benchmark target exists in the candidate image region by means of the enhanced image features during subsequent processing, ultimately improving the accuracy of target detection. The enhanced image features of each supporting image are used as class prototypes, thus making the class prototypes more diverse, without losing the diversity of class prototypes within the same category, which helps to improve the accuracy of target detection. The target detection device averages the enhanced image features of the shared information, which have more diverse and detailed characteristics, resulting in more accurate average image features, i.e., class prototypes, thus maintaining a high level of target detection accuracy.

[0213] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A target detection method, characterized in that, The method includes: From the image to be detected, features of the image to be detected are extracted, and from at least one support image of the reference target, at least one support image feature is extracted respectively; the support image is an image labeled with the location information of the reference target. Enhancement of shared information is performed on each of the supporting image features to obtain enhanced image features corresponding to each supporting image; wherein, the shared information represents the commonality of the benchmark target in at least one of the supporting images, and the shared information is obtained by processing the mapping vectors of different feature spaces using an attention mechanism, and the mapping vectors are obtained by mapping each supporting image feature to the different feature spaces; Based on the features of the image to be detected, at least one candidate image region is determined from the image to be detected; The enhanced image features of at least one of the supporting images are averaged to obtain average image features; Calculate the feature similarity between the average image features and the candidate image features corresponding to each candidate image region; Based on the feature similarity, the target image region where the reference target is located is determined from at least one of the candidate image regions.

2. The method according to claim 1, characterized in that, The enhancement of shared information for each of the supporting image features to obtain the enhanced image features corresponding to each supporting image includes: At least one of the supporting image features is concatenated to obtain concatenated supporting features; The stitched support features are subjected to multiple feature extractions to obtain multiple mapped feature vectors; each mapped feature vector contains a feature component corresponding to each of the support image features. Based on multiple mapping feature vectors, the shared information in each of the supporting image features is enhanced to obtain the enhanced image features corresponding to each of the supporting images.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Based on the enhanced image features of each of the supporting images, a first attention weight is determined for the candidate image features corresponding to each candidate image region; the first attention weight describes the similarity between the image content contained in the candidate image region and the benchmark target. The candidate image features are weighted using the first attention weight to obtain weighted image features for each candidate image region; Based on the weighted image features of each candidate image region, the target image region where the reference target is located is determined from at least one of the candidate image regions.

4. The method according to claim 3, characterized in that, Determining the target image region where the reference target is located from at least one of the candidate image regions based on the weighted image features of each of the candidate image regions includes: Based on the weighted image features of each candidate image region, determine the image category and confidence level of each candidate image region; At least one candidate image region that meets the target conditions is determined as the target image region where the reference target is located; The target condition is that the image category matches the benchmark target, and the confidence level is greater than the confidence level threshold.

5. The method according to claim 3, characterized in that, The step of determining the first attention weight of the candidate image features corresponding to each candidate image region based on the enhanced image features of each of the supporting images includes: The enhanced image features of each of the supporting images are stitched together to obtain stitched enhanced features, and the candidate image features of each of the candidate image regions are stitched together to obtain stitched candidate features; At least two feature extractions are performed on the splicing enhancement features to obtain at least two enhancement feature vectors, and feature extraction is performed on the splicing candidate features to obtain candidate feature vectors; the candidate feature vectors include feature components corresponding to the candidate image features of each candidate image region; By applying a multi-head attention mechanism to at least two of the enhanced feature vectors and the candidate feature vectors, the attention weight of each feature component of the candidate feature vector is obtained; The attention weight of each feature component of the candidate feature vector is determined as the first attention weight of the candidate image feature corresponding to the candidate image region.

6. The method according to claim 1 or 2, characterized in that, After enhancing each of the supporting image features with shared information to obtain the enhanced image features corresponding to each supporting image, before averaging the enhanced image features of at least one supporting image to obtain the average image features, the method further includes: Based on the enhanced image features corresponding to each of the supporting images, a second attention weight is determined for each sub-image feature of the image to be detected; Using the second attention weight, each of the sub-image features is weighted to obtain weighted sub-image features; Based on each of the weighted sub-image features, at least one of the candidate image regions is determined from the image to be detected.

7. The method according to claim 1 or 2, characterized in that, The step of extracting at least one support image feature from at least one support image of the benchmark target includes: Feature extraction is performed on the labeled position of the reference target in at least one of the supporting images to obtain at least one target position feature; Feature extraction is performed on the image content of at least one of the supporting images to obtain at least one initial image feature; At least one of the target location features is fused with at least one of the initial image features to obtain at least one of the supporting image features.

8. The method according to claim 1 or 2, characterized in that, After performing shared information enhancement on each of the supporting image features to obtain the enhanced image features corresponding to each supporting image, the method further includes: The enhanced image features corresponding to each of the supporting images are stored; the enhanced image features are also used to determine the latest target image region where the benchmark target is located from at least one latest candidate image region of the latest image to be detected.

9. The method according to claim 1 or 2, characterized in that, The benchmark target includes: a virtual object; after determining the target image region where the benchmark target is located from at least one of the candidate image regions based on the feature similarity, the method further includes: Control operations are generated for the virtual object; the control operations are used to control the virtual object to complete target actions in the virtual scene, the target actions include: moving, releasing skills, and interacting with virtual props.

10. The method according to claim 1 or 2, characterized in that, From extracting the features of the image to be detected from the image to be detected, to determining the target image region where the reference target is located from at least one of the candidate image regions based on the feature similarity, all of these are achieved through a target detection model. Before extracting the features of the image to be detected from the image to be detected, the method further includes: Obtain first training data and second training data; wherein the number of labeled data in the first training data is greater than the number of labeled data in the second training data; Using the first training data, the initial detection model is trained to obtain an intermediate detection model; Candidate training data is extracted from the second training data; the candidate training data includes K labeled training instances, wherein the labeled training instances refer to images that have the location information of the reference target labeled; The intermediate detection model is trained using the target training data composed of the first training data and the candidate training data to obtain the target detection model.

11. A target detection device, characterized in that, The device includes: The feature extraction module is used to extract features of the image to be detected from the image to be detected, and to extract features of at least one support image from at least one support image of the reference target; the support image is an image labeled with the location information of the reference target. The feature enhancement module is used to enhance the shared information for each of the supporting image features to obtain the enhanced image features corresponding to each supporting image; wherein, the shared information represents the commonality of the benchmark target in at least one of the supporting images, and the shared information is obtained by processing the mapping vectors of different feature spaces using an attention mechanism, and the mapping vectors are obtained by mapping each supporting image feature to the different feature spaces; A candidate generation module is used to determine at least one candidate image region from the image to be detected based on the features of the image to be detected; The target determination module is used to perform average processing on the enhanced image features of at least one of the supporting images to obtain average image features; calculate the feature similarity between the average image features and the candidate image features corresponding to each candidate image region; and determine the target image region where the reference target is located from at least one of the candidate image regions based on the feature similarity.

12. The apparatus according to claim 11, characterized in that, The feature enhancement module is further configured to perform feature stitching on at least one of the supporting image features to obtain stitched supporting features; The stitched support features are subjected to multiple feature extractions to obtain multiple mapped feature vectors; each mapped feature vector contains feature components corresponding to each of the support image features; based on the multiple mapped feature vectors, the shared information in each of the support image features is enhanced to obtain the enhanced image features corresponding to each support image.

13. The apparatus according to claim 11, characterized in that, The target determination module is further configured to determine a first attention weight of the candidate image features corresponding to each candidate image region based on the enhanced image features of each of the supporting images; The first attention weight describes the similarity between the image content contained in the candidate image region and the benchmark target; the candidate image features are weighted using the first attention weight to obtain the weighted image features of each candidate image region; based on the weighted image features of each candidate image region, the target image region where the benchmark target is located is determined from at least one candidate image region.

14. A target detection device, characterized in that, The target detection device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the target detection method according to any one of claims 1 to 10.

15. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the target detection method according to any one of claims 1 to 10.

16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the target detection method according to any one of claims 1 to 10.

Images