Target detection method and its device, equipment, medium, and product

Abstract

The present application discloses a target detection method and its device, equipment, medium, and product, the method comprising: obtaining an advertising picture to be detected; using a target detection model that has been trained to convergence to perform target detection on the advertising picture, and intercepting the target area image in the advertising picture according to the detection result; then using an image recognition model that has been trained to convergence to perform image recognition on the advertising picture and the target area image respectively, and obtain corresponding recognition result combinations to form a target recognition sequence; finally, identifying the actual needs of the instance application scenario, obtaining the probability value under preset conditions from the target recognition sequence according to the actual needs as the final probability value of whether the advertising picture contains the target item, making a final judgment, and outputting the result. The present application can efficiently screen and enhance the data of the relevant pictures that constitute the above-mentioned model training set, so as to be used for training relevant models, making the relevant models more accurate and widely adaptable.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a target detection method and its corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of technology, the use of artificial neural network models for object detection has become mainstream. Application scenarios such as e-commerce platforms generate massive amounts of advertising images daily. Due to the impact of risk control measures, advertising messages posted on e-commerce platforms must comply with relevant regulations. Due to restrictions on sensitive items in advertising images, such as flags, e-commerce platforms generally require object detection and recognition in user-uploaded advertising images before further processing.

[0003] For application scenarios such as e-commerce platforms, if the advertising images uploaded by merchants to display advertising information contain sensitive objects, the current main practice is to conduct manual screening, which is time-consuming and labor-intensive, and is prone to missed detections and false detections due to manual fatigue. Secondly, the application of existing target recognition models and target detection models in related application examples also has the problem of high false detection rates or high missed detection rates. For e-commerce platforms, this will cause advertisements posted by users that comply with relevant regulations to be mistakenly detected as containing sensitive objects, or advertisements posted by users that do not comply with relevant regulations to be missed and detected as normal, which will cause great trouble for e-commerce platforms.

[0004] Furthermore, relying solely on manual investigation is unrealistic and time-consuming. Nowadays, products on e-commerce platforms are rapidly evolving, and so are their corresponding advertising information. Therefore, e-commerce users must be able to quickly update their advertising information in order to compete for sales share in the e-commerce market.

[0005] Therefore, how to accurately and efficiently identify target objects from advertisement pictures to be detected that contain various sensitive objects and make the recognition results more accurate has become a technical problem that needs to be solved in this field. Summary of the Invention

[0006] The primary purpose of the present application is to solve at least one of the above problems and to provide a target detection method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product.

[0007] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0008] A target detection method includes the following operations: obtaining an advertisement image to be detected;

[0009] Performing target detection on the advertisement image using a target detection model that has been trained to convergence to obtain a target area image;

[0010] Using an image recognition model that has been trained to convergence to perform image recognition on the advertisement image and the target area image, respectively, to obtain multiple recognition results to form a target recognition sequence;

[0011] The target recognition sequence is used to output corresponding target recognition results based on the preset conditions of a specific instance scenario.

[0012] In a further embodiment, obtaining an advertisement image to be detected includes the following steps:

[0013] Responding to a user-triggered advertisement publishing request and obtaining the corresponding submitted advertisement publishing information, wherein the advertisement publishing information includes an advertisement image;

[0014] Acquire the advertisement picture from the advertisement publishing information.

[0015] In a further embodiment, target detection is performed on the advertisement image using a target detection model that has been trained to convergence to obtain a target area image, including the following steps:

[0016] Using a convolutional backbone network to extract features from the advertisement image, obtaining multi-layer feature maps of different scales;

[0017] A region generation network is used to generate multiple candidate regions of interest for the multi-layer feature maps of different scales, and then an alignment operation of the regions of interest is performed;

[0018] The head network is used to perform three-branch processing on the feature map after alignment of the region of interest, namely bounding box regression processing, recognition processing and mask map prediction to obtain the detection result;

[0019] According to the detection result, the target area in the advertisement image is intercepted to obtain a corresponding target area image.

[0020] In a further embodiment, an image recognition model that has been trained to convergence is used to perform image recognition on the advertisement image and the target area image respectively, and multiple recognition results are obtained to form a target recognition sequence, including the following steps:

[0021] Perform image block embedding on the input image to obtain multiple image block vectors, and add a classification vector to form multiple embedding vectors;

[0022] Adding a position encoding vector to the multiple embedding vectors to form an input vector, wherein the position encoding vector can maintain spatial position information between image blocks;

[0023] Stacking multiple encoding modules for the input vector to extract features, wherein the encoding modules include multi-head attention and multi-layer perceptron;

[0024] A multi-layer perceptron is used to further transform the classification space of the final deep classification vector to obtain its classification probability;

[0025] The above steps are performed on the advertisement picture and the target area image respectively to obtain corresponding multiple recognition results, which constitute a target recognition sequence of the advertisement picture.

[0026] In a further embodiment, the target recognition sequence is used to output a corresponding target recognition result according to a preset condition of a specific instance scenario, including the following steps:

[0027] Sorting the target recognition sequence to obtain its maximum and minimum values;

[0028] Identify the actual needs of the application scenario, using the minimum value as the probability value of whether the advertisement image contains the target item in a high-accuracy scenario, and the maximum value as the probability value of whether the advertisement image contains the target item in a high-recall scenario;

[0029] The probability value is compared with a preset threshold value. When the probability value is greater than the preset threshold value, it is determined that the advertisement image contains the target item; otherwise, it is determined that the advertisement image does not contain the target item.

[0030] In a preferred embodiment, the target detection model is a Mask-RCNN model, the image recognition model is a ViT model, the basic network architecture of the target detection model is a Mask-RCNN model, the basic network architecture of the image recognition model is a ViT model, and the target object is a flag.

[0031] A target detection device provided to meet one of the purposes of the present application includes an image acquisition module, a target detection module, an image recognition module and a target discrimination module, wherein the image acquisition module is used to acquire an advertising picture to be detected; the target detection module is configured to use a target detection model that has been trained to convergence to perform target detection on the advertising picture to obtain a target area image; the image recognition module is configured to use an image recognition model that has been trained to convergence to perform image recognition on the advertising picture and the target area image respectively, obtain multiple recognition results, and constitute a target recognition sequence; the target discrimination module is used to use the target recognition sequence to output corresponding target recognition results according to preset conditions of a specific instance scenario.

[0032] In a further embodiment, the image acquisition module includes: a response submodule, used to respond to the advertising publishing request triggered by the user and obtain the corresponding advertising publishing information submitted, wherein the advertising publishing information includes advertising pictures; an acquisition submodule; used to obtain the advertising pictures from the advertising publishing information, for identifying the pictures of the target objects to be detected.

[0033] In the in-depth example, the target detection module includes: a convolutional backbone sub-module, which uses a convolutional backbone network to perform feature extraction on the advertising image to obtain multiple layers of feature maps of different scales; a region of interest sub-module, which uses a region generation network to generate multiple candidate regions of interest for the multiple layers of feature maps of different scales, and then performs a region of interest alignment operation; a detection sub-module, which uses a head network to perform three branch processing on the feature map after region of interest alignment, namely, bounding box regression processing, recognition processing and mask map prediction, to obtain a detection result; a clipping sub-module, which is used to clip the target area in the advertising image according to the detection result to obtain the corresponding target area image.

[0034] In a further example, the image recognition module includes an embedding submodule, which is used to embed image blocks for the input image, obtain multiple image block vectors, and add a classification vector to form multiple embedding vectors; a position encoding submodule, which is used to add position encoding vectors to the multiple embedding vectors to form an input vector, and the position encoding vector can maintain the spatial position information between image blocks; a feature extraction submodule, which is used to stack multiple encoding modules for feature extraction on the above-mentioned input vector, and the encoding module includes multi-head attention and multi-layer perceptron; a classification submodule, which uses a multi-layer perceptron to perform further classification space transformation on the final deep classification vector to obtain its classification probability; a sequence generation submodule, which performs the above steps on the advertising picture and the target area image respectively to obtain corresponding multiple recognition results, forming a target recognition sequence of the advertising picture.

[0035] In the further example, the target discrimination module includes: a sorting submodule, which is used to sort the target recognition sequence and obtain its maximum and minimum values; a probability value calculation submodule, which is used to identify the actual needs of the example application scenario, and in a high-accuracy scenario, the minimum value is used as the probability value of whether the advertising image contains the target item, and in a high-recall scenario, the maximum value is used as the probability value of whether the advertising image contains the target item; a discrimination submodule, which is used to compare the probability value with a preset threshold value, and when the probability value is greater than the preset threshold value, it is determined that the advertising image contains the target item, otherwise it is determined that it does not contain the target item.

[0036] In a preferred embodiment, the target detection model is a Mask-RCNN model, the image recognition model is a ViT model, the basic network architecture of the target detection model is a Mask-RCNN model, the basic network architecture of the image recognition model is a ViT model, and the target object is a flag.

[0037] A computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the target detection method described in the present application.

[0038] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the target detection method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.

[0039] A computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.

[0040] Compared with the prior art, the advantages of this application are as follows:

[0041] The present application obtains an advertising picture to be detected; uses a target detection model that has been trained to convergence to perform target detection on the advertising picture, and captures a target area image in the advertising picture based on the detection result; then uses an image recognition model that has been trained to convergence to perform image recognition on the advertising picture and the target area image respectively, thereby obtaining corresponding recognition results, and combining the recognition results to form a target recognition sequence, which reflects multiple probability values ​​of the target item contained in the advertising picture; finally, the actual needs of the identification instance application scenario are identified, and according to the actual needs, the probability value under preset conditions is obtained from the target recognition sequence as the final probability value of whether the advertising picture contains the target item, and a final judgment is made to output the result.

[0042] This application adopts enhanced detection of detection, recognition and scene conditions. That is, a target detection model is used to perform target detection on the advertising image to obtain a target area image containing the target object in the advertising image; image recognition is performed on the advertising image and the target area image respectively to obtain a target recognition sequence; different probability values ​​are selected according to different instance application scenarios as the final probability estimate value of whether the advertising image contains the target object for judgment. On the one hand, it greatly improves the accuracy of target object recognition, helps e-commerce platforms to more effectively exclude advertising images that violate relevant regulations, and greatly reduces time and labor costs; on the other hand, the implementation of the target recognition sequence can well solve the business needs of high accuracy and high recall in the instance application scenarios of e-commerce platforms.

[0043] To sum up, the judgment result made by this application on whether the target object is contained in the advertising image has a higher confidence level and can be highly trusted. It is suitable for detecting whether there are sensitive target objects in advertising images in application scenarios such as e-commerce platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0045] Figure 1 A flowchart of a typical embodiment of the target detection method of the present application is shown;

[0046] Figure 2 This is a schematic diagram of the process of performing target detection on an advertising image in an embodiment of the present application;

[0047] Figure 3 Schematic diagram of the process of performing image recognition on an input image in an embodiment of the present application;

[0048] Figure 4 This is a flow chart of outputting target recognition results for an example scenario in an embodiment of the present application;

[0049] Figure 5 This is a principle block diagram of the target detection device of this application;

[0050] Figure 6 This is a schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0051] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0052] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0054] It will be understood by those skilled in the art that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, the terms "client," "terminal," or "terminal device" may be portable, transportable, or installed in a vehicle (air, sea, and / or land), or may be adapted and / or configured to operate locally and / or in a distributed manner at any other location on Earth and / or in space. As used herein, the terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or may include a smart TV, a set-top box, or other device.

[0055] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0056] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0057] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0058] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0059] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.

[0060] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0061] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.

[0062] A target detection method of the present application can be programmed as a computer program product and deployed in a client or server for execution. The method can be executed by accessing an interface opened after the computer program product is running and conducting human-computer interaction with the process of the computer program product through a graphical user interface.

[0063] See also Figure 1 In a typical embodiment, the target detection method of the present application includes the following steps:

[0064] Step S1100: Obtain an advertisement image to be detected;

[0065] In an exemplary application scenario used to assist in explanation, the advertising image to be detected may be an advertising image containing items on an e-commerce platform. The items displayed in the image are generally non-sensitive items. The implementation of this application is to identify sensitive items from the advertising image, that is, pre-set, sensitive target items, such as flags, etc.; therefore, it is necessary to identify whether sensitive items are contained in the advertising image for further processing.

[0066] In the application scenario of the e-commerce platform, if it is necessary to obtain the pictures to be detected, one implementation method is to receive the input of the e-commerce platform users, especially the input when the merchant instance users configure the advertising information, and use the advertising pictures in the advertising information as the pictures to be detected; in another way, the server of the e-commerce platform can process the advertising pictures in the e-commerce platform database in batches in the background, and use these advertising pictures as the pictures to be detected for target detection.

[0067] Step S1200: Performing target detection on the advertisement image using a target detection model that has been trained to convergence to obtain a target area image;

[0068] The target detection model is a model that takes an advertising image as input and takes the target object label and its bounding box indicating the image location of the target area as output.

[0069] The target detection model is preferably implemented as a neural network model. For example, in the embodiment of the present application, the target detection model is a Mask-RCNN that has been trained to convergence. Alternatively, the neural network model can be selected from a variety of relatively excellent target detection models in the existing technology, including but not limited to: YOLO series models, other R-CNN series models, SSD models, DETR, etc., all of which are mature target detection models.

[0070] In this embodiment, the target detection model that has been trained to convergence includes three network components. The first network component is a convolutional backbone network, which is used to extract features from the advertising image and obtain multiple layers of feature maps at different scales. The second network component is a region generation network, which is used to generate multiple candidate regions of interest for the multiple layers of feature maps at different scales and then perform region of interest alignment operations. The third network component is a head network, which is used to perform three branch processing on the feature map after region of interest alignment, namely, bounding box regression processing, recognition processing, and mask map prediction, to obtain detection results.

[0071] The detection result includes a bounding box indicating the area where the target item in the advertisement image is located. Based on the bounding box, the target area indicated in the advertisement image is captured to obtain a corresponding target area image. The number of bounding boxes depends on the detection results in the actual application scenario and may be none, one, or multiple. Accordingly, the number of target area images matches the number of bounding boxes. The advertisement image and the target area images are output to the next step for further processing.

[0072] Step S1300: using an image recognition model that has been trained to convergence to perform image recognition on the advertisement image and the target area image, respectively, to obtain multiple recognition results, forming a target recognition sequence;

[0073] The image recognition model is a model that takes the advertisement picture or the target area image as input and outputs a probability value of containing the target item.

[0074] In the embodiment of the present application, the image recognition model is a ViT that has been trained to convergence. The image recognition model is preferably implemented as a neural network model. Alternatively, the neural network model can be selected from a variety of relatively excellent image recognition models in the existing technology, including but not limited to: VGG series models, Inception series models, ResNet series models, EfficientNet series models, HRNet, etc., all of which are mature image recognition models. As long as they are trained to convergence with sufficient corresponding training samples, they can be used as the image recognition model of the present application.

[0075] The image recognition model includes three components. The first component is an embedding component, which is used to embed image blocks for the input image, obtain multiple image block vectors, and add a classification vector to form multiple embedding vectors. Then, a position encoding vector is added to the multiple embedding vectors to form an input vector. The position encoding vector can maintain the spatial position information between image blocks; the second component is a feature extraction component, which is used to stack multiple encoding modules for feature extraction on the above input vector. The encoding module includes multi-head attention and multi-layer perceptron; the third component is a classification component, which is configured to use a multi-layer perceptron to perform further classification space transformation on the final deep classification vector to obtain its classification probability;

[0076] The image recognition model is used to perform the above-mentioned image recognition operation on the advertising image and the target area image obtained in the previous step, respectively, to obtain corresponding multiple recognition results, that is, a recognition result is generated for each input image, that is, the probability value of whether the target item is contained, thereby forming a target recognition sequence for the advertising image.

[0077] Step S1400: using the target recognition sequence to output a corresponding target recognition result according to preset conditions of a specific instance scenario;

[0078] The target recognition sequence includes multiple probability values ​​for whether the advertising image contains the target item. The target recognition sequence is sorted to obtain its maximum and minimum values; then, the actual needs of the instance application scenario are identified, and the minimum value is used as the probability value of whether the advertising image contains the target item in a high-accuracy scenario, and the maximum value is used as the probability value of whether the advertising image contains the target item in a high-recall scenario; finally, the probability value is compared with a preset threshold. When the probability value is greater than the preset threshold, it is determined that the advertising image contains the target item; otherwise, it is determined that the target item is not contained.

[0079] In summary, this typical embodiment shows that the present application obtains an advertising picture to be detected; uses a target detection model that has been trained to convergence to perform target detection on the advertising picture, and captures the target area image in the advertising picture based on the detection result; then uses an image recognition model that has been trained to convergence to perform image recognition on the advertising picture and the target area image respectively, and then obtains corresponding recognition results, and combines the recognition results to form a target recognition sequence, which reflects multiple probability values ​​of the target item contained in the advertising picture; finally, the actual needs of the identification instance application scenario are identified, and according to the actual needs, the probability value under preset conditions is obtained from the target recognition sequence as the final probability value of whether the advertising picture contains the target item, and a final judgment is made to output the result.

[0080] This application adopts enhanced detection of detection, recognition and scene conditions. That is, a target detection model is used to perform target detection on the advertising image to obtain a target area image containing the target object in the advertising image; image recognition is performed on the advertising image and the target area image respectively to obtain a target recognition sequence; different probability values ​​are selected according to different instance application scenarios as the final probability estimate value of whether the advertising image contains the target object for judgment. On the one hand, it greatly improves the accuracy of target object recognition, helps e-commerce platforms to more effectively exclude advertising images that violate relevant regulations, and greatly reduces time and labor costs; on the other hand, the implementation of the target recognition sequence can well solve the business needs of high accuracy and high recall in the instance application scenarios of e-commerce platforms.

[0081] To sum up, the judgment result made by this application on whether the target object is contained in the advertising image has a higher confidence level and can be highly trusted. It is suitable for detecting whether there are sensitive target objects in advertising images in application scenarios such as e-commerce platforms.

[0082] See also Figure 2 In a further embodiment, step S1200, performing target detection on the advertisement image using a target detection model that has been trained to convergence to obtain a target area image, includes the following steps:

[0083] Step S1210: extract features from the advertisement image using a convolutional backbone network to obtain multi-layer feature maps of different scales;

[0084] The convolutional backbone network adopts the ResNet-FPN architecture, specifically consisting of two parts: the first, using ResNet-50 as the backbone network for bottom-up feature extraction, and the second, using the FPN structure, or feature pyramid network, to deliver strong semantic information from top to bottom. ResNet-FPN fuses features from all levels, imparting both strong semantic and spatial information, thereby enhancing the semantic and positional representation of the feature maps at different scales.

[0085] The Resnet-50 skeleton network performs feature extraction on the advertisement image, and can be divided into five stages according to the size of the extracted feature map, wherein the feature layers output by stages 1, 2, 3, 4, and 5 are defined as C1, C2, C3, C4, and C5.

[0086] The FPN structure adopts a top-down structure with horizontal connections, fusing feature maps from shallow to deep layers, fully utilizing the semantic and positional features at each stage. The FPN structure performs a 1x1 convolution on the topmost layer C5 of the backbone network to obtain a feature layer P5 with a preset number of channels. P5 is then upsampled to obtain feature layer 1, which has the same resolution as feature layer C4. The next feature layer C4 undergoes a 1x1 convolution to obtain feature layer 2 with a preset number of channels. Feature layer 1 and feature layer 2 are added and fused to obtain feature layer P4. Similarly, feature layers C5, C4, C3, and C2 are used to obtain feature layers P5, P4, P3, and P2. Furthermore, P5 undergoes further feature extraction to obtain feature layer P6. P2-P6 are used for the region generation network component, and P2-P5 are used for the head network component.

[0087] The larger feature maps obtained through upsampling lack edge detail information, and the feature maps obtained through pooling will inevitably lose some edge features. The ResNet-FPN simultaneously utilizes the high resolution of shallow feature maps and the high semantic information of deep feature maps. Fusing these feature maps from different layers is equivalent to fusing strong semantic information and strong edge information, thereby improving feature extraction.

[0088] Step S1220: using a region generation network to generate multiple candidate regions of interest for the multiple layers of feature maps at different scales, and then performing a region of interest alignment operation;

[0089] The region generation network uses a sliding window to slide the four feature maps P2-P6 one by one, initializing a reference region at each point in the sliding window. This means that the specific coordinates of each base anchor box are calculated based on the coordinates of the sliding window. 1k base anchor boxes are generated on each feature layer. For each base anchor box, two confidence scores are generated: one for the foreground and one for the background. Four coordinate deviation regression values ​​are also generated.

[0090] The region of interest alignment operation RolAlign is an improvement to the region of interest pooling RolPool; the RolPool uses two rounding operations when restoring the feature map to the original image size, which will lead to deviation results for multiple pixel points; while RolAlign directly cancels the rounding operation and obtains the pixel values ​​of the fixed four point coordinates through bilinear interpolation, reducing the restoration error.

[0091] The region generation network can generate multiple candidate regions of interest and is equipped with RolAlign to achieve alignment operations on the regions of interest.

[0092] Step S1230: Use the head network to perform three branch processing on the feature map after alignment with the region of interest, namely, bounding box regression processing, recognition processing and mask map prediction to obtain a detection result.

[0093] The head network performs final detection on the aligned feature maps. This detection is divided into three branches. The bounding box regression branch and the recognition branch use convolutional layers with a depth of 1024 instead of fully connected layers for prediction, which can more fully utilize feature information. The mask prediction branch uses a fully convolutional network (FCN) to achieve semantic segmentation. The mask map for each ROI has 80 categories, which can reduce competition between categories and achieve better results.

[0094] Therefore, the detection results of the head network include the confidence of the target object, and the bounding box and mask map of the target area image indicated by it.

[0095] Step S1240: According to the detection result, the target area in the advertisement image is intercepted to obtain a corresponding target area image.

[0096] The detection results include a bounding box, mask, and confidence level indicating the target area image within the advertisement image containing the target item. In this embodiment, only the bounding box information in the detection results is used. Based on the target area image locations indicated by the bounding box information, the same number of target area images is captured from the advertisement image. This number can be 0, 1, or more in a specific application scenario, depending on the specific value obtained in the application.

[0097] The target area image and the advertisement picture are output together to step S1300 for further processing.

[0098] In this embodiment, the target detection model adopts the Mask-RCNN model and uses the ResNet-FPN structure, while fusing the strong semantic information of deep features and the strong edge information of shallow features; the RolAlign operation is used to achieve pixel alignment in the generation of the region of interest to reduce the position deviation estimation of the bounding box, thereby effectively enhancing the detection effect of the target.

[0099] See also Figure 3 In the further example, step S1300, using the image recognition model that has been trained to convergence to perform image recognition on the advertisement image and the target area image respectively, obtaining multiple recognition results to form a target recognition sequence, includes the following steps:

[0100] Step S1310: performing image block embedding on the input image to obtain multiple image block vectors, and simultaneously adding a classification vector to form multiple embedding vectors;

[0101] The image recognition model is a ViT that has been trained to convergence. The model is a model that is further improved based on the Transformer model applied to NLP problems and acts on the image field. Its first step is to convert images in the image field into word structures in natural language processing. This step is called image block embedding. Specifically, the picture is standardized to obtain a picture of standard size, which can be regarded as a complete sentence; then it is divided into small blocks of fixed size, called Patches, and the pixel values ​​of each small block are flattened to become a word in the sentence. Subsequently, each Patch is compressed into a vector of a certain dimension through a fully connected network, thereby obtaining multiple image block vectors. This process is image block embedding, called Patch Embedding. In the example application scenario of the present application, the standard size is 224*224, the fixed size of Patch is 16*16, and the dimension of the vector is 768.

[0102] After obtaining multiple image block vectors, a classification vector of the same dimension is spliced ​​together. As the name suggests, this vector is used to learn category information during the model training process. The classification vector is a learnable embedding vector.

[0103] Step S1320: Add a position encoding vector to the multiple embedding vectors to form an input vector, where the position encoding vector can maintain spatial position information between image blocks;

[0104] Multiple embedding vectors are obtained from step S1310, but these embedding vectors lack position information. That is, except for the classification vector, all other vectors have corresponding position information in the image. Therefore, to maintain the spatial position information between each patch in the input image, it is necessary to add a position encoding vector to these multiple embedding vectors. Specifically, a one-dimensional learnable position embedding vector is directly used and added to the embedding vector to form the input vector.

[0105] Therefore, the input image has completed the vector embedding work here and can be put into the Transformer for training and feature learning.

[0106] Step S1330: stack multiple encoding modules for the input vector to perform feature extraction, wherein the encoding modules include a multi-head attention and a multi-layer perceptron;

[0107] Multiple encoding modules are stacked for the above input vector. The encoding modules mainly include multi-head attention and multi-layer perceptron. Specifically, the encoding module consists of two parts. The first part is layer normalization (Layer Norm) -> multi-head attention (Multi-Head Attention) -> dropout -> short cut. The second part is layer normalization (Layer Norm) -> multi-layer perceptron (Multi-Layer Perception) -> dropout -> short cut. The multi-layer perceptron includes full connection (Linear) -> activation function (GELU) -> dropout -> full connection (Linear) -> dropout.

[0108] The layer normalization is to perform normalization on a specified dimension of a single data.

[0109] The feature extraction enables the classification vector to fuse the semantic features of all image blocks.

[0110] Step S1340: Using a multilayer perceptron to perform further classification space transformation on the final deep classification vector to obtain its classification probability;

[0111] After going through the multiple encoding modules in the above steps, the classification vector has extracted the feature information of each image block in the image. The classification vector is then put into a multi-layer perceptron for the final classification space transformation. The multi-layer perceptron includes layer normalization (Layer Norm) -> fully connected (Linear).

[0112] This step ultimately obtains the probability value of whether the input image contains the target object, which is called the classification probability.

[0113] Step S1350 : Perform the above steps on the advertisement image and the target area image respectively to obtain corresponding multiple recognition results, forming a target recognition sequence of the advertisement image.

[0114] The above steps, i.e., image recognition model processing, are performed on the advertising image and the target area image, respectively, to obtain multiple corresponding classification probabilities, one for each input image. These multiple classification probabilities are combined to form a sequence, called a target recognition sequence, which indicates whether the advertising image contains the target item and is used for further processing in the next step.

[0115] The embodiment of the present application adopts ViT as the network architecture of the image recognition model, divides the image into image blocks, completes image block embedding and position code embedding, and then uses the strong semantic feature extraction method of Transformer to extract the semantic features of the image. Finally, the final classification probability prediction is achieved through further classification space transformation of the classification vector, so that the image recognition effect is significantly better than other image recognition models, thereby obtaining better detection effects.

[0116] See also Figure 4 In the in-depth example, step S1400, using the target recognition sequence to output the corresponding target recognition result according to the preset conditions of the specific instance scene, includes the following steps:

[0117] Step S1410: sort the target recognition sequence to obtain its maximum and minimum values;

[0118] The target recognition sequence includes multiple classification probabilities, each of which has a value between 0 and 1. The target recognition sequence is directly composed of multiple probability values ​​obtained by performing image recognition on multiple input images, and its arrangement is random. Therefore, the target recognition sequence is first sorted so that the classification probabilities in the sequence are arranged from large to small. Then, its maximum and minimum values ​​are obtained:

[0119] OutputProb_min=min(OutputVitList)

[0120] OutputProb_max=max(OutputVitList)

[0121] Among them, OutputVitList represents the target recognition sequence; OutputProb_min represents the minimum value in the target recognition sequence; OutputProb_max represents the maximum value in the target recognition sequence.

[0122] Step S1420: Identify the actual requirements of the example application scenario, use the minimum value as the probability value of whether the advertisement image contains the target item in a high-accuracy scenario, and use the maximum value as the probability value of whether the advertisement image contains the target item in a high-recall scenario;

[0123] The actual needs of the identification instance application scenario are different in different application scenarios. Specifically, in the embodiment of the present application, in the instance application scenario where high accuracy is required, the minimum value in the target recognition sequence is used as the final probability value of whether the advertising image contains the target item; in the instance application scenario where high recall is required, the maximum value in the target recognition sequence is used as the final probability value of whether the advertising image contains the target item.

[0124] Step S1430: Compare the probability value with a preset threshold value. When the probability value is greater than the preset threshold value, determine that the advertisement image contains the target item; otherwise, determine that the advertisement image does not contain the target item.

[0125] Based on the previous step, a probability value for the example application scenario is obtained. This probability value is compared with a preset threshold. If the probability value is greater than the preset threshold, the advertisement image is determined to contain the target item. If the probability value is less than the preset threshold, the advertisement image is determined to not contain the target person. In the example application scenario, further business processing can be performed on the advertisement image based on the determination result.

[0126] The preset threshold is the dividing value for determining whether the probability value indicates that the advertising image contains the target item. The setting of this value directly affects the accuracy of the determination. If it is set too high, an advertising image that does not contain the target item will be determined to contain it. If it is set too low, an advertising image that contains the target item will be determined not to contain it. Therefore, the threshold needs to be set by relevant technical personnel based on comparison of test results and application of prior knowledge.

[0127] This embodiment uses the obtained target recognition sequence to select different probability values ​​based on different application scenarios as the final probability estimate for whether the advertising image contains the target item, and then makes a final determination. This step can effectively improve the accuracy of target item recognition on e-commerce platforms under specific application scenarios and business needs, helping them more effectively exclude advertising images that violate relevant regulations.

[0128] See also Figure 5 , a target detection device provided to meet one of the purposes of the present application includes an image acquisition module 1100, a target detection module 1200, an image recognition module 1300 and a target discrimination module 1400, wherein the image acquisition module 1100 is used to acquire an advertising picture to be detected; the target detection module 1200 is configured to use a target detection model that has been trained to convergence to perform target detection on the advertising picture to obtain a target area image; the image recognition module 1300 is configured to use an image recognition model that has been trained to convergence to perform image recognition on the advertising picture and the target area image respectively, obtain multiple recognition results, and constitute a target recognition sequence; the target discrimination module 1400 is used to use the target recognition sequence to output corresponding target recognition results according to preset conditions of a specific instance scenario.

[0129] In a further embodiment, the image acquisition module 1100 includes: a response submodule, used to respond to the advertising publishing request triggered by the user, and obtain the corresponding advertising publishing information submitted, wherein the advertising publishing information includes advertising pictures; an acquisition submodule; used to obtain the advertising pictures from the advertising publishing information, and used to identify the pictures of the target objects to be detected.

[0130] In the in-depth example, the target detection module 1200 includes: a convolutional backbone sub-module, which uses a convolutional backbone network to perform feature extraction on the advertising image to obtain multiple layers of feature maps of different scales; a region of interest sub-module, which uses a region generation network to generate multiple candidate regions of interest for the multiple layers of feature maps of different scales, and then performs a region of interest alignment operation; a detection sub-module, which uses a head network to perform three branch processing on the feature map after region of interest alignment, namely, bounding box regression processing, recognition processing and mask map prediction, to obtain a detection result; a clipping sub-module, which is used to clip the target area in the advertising image according to the detection result to obtain a corresponding target area image.

[0131] In the further example, the image recognition module 1300 includes an embedding submodule, which is used to embed image blocks for the input image, obtain multiple image block vectors, and add a classification vector to form multiple embedding vectors; a position coding submodule, which is used to add position coding vectors to the multiple embedding vectors to form an input vector, and the position coding vector can maintain the spatial position information between image blocks; a feature extraction submodule, which is used to stack multiple coding modules for feature extraction on the above-mentioned input vector, and the coding module includes multi-head attention and multi-layer perceptron; a classification submodule, which uses a multi-layer perceptron to perform further classification space transformation on the final deep classification vector to obtain its classification probability; a sequence generation submodule, which performs the above steps on the advertising picture and the target area image respectively to obtain corresponding multiple recognition results, forming a target recognition sequence of the advertising picture.

[0132] In the further example, the target discrimination module 1400 includes: a sorting submodule, which is used to sort the target recognition sequence and obtain its maximum and minimum values; a probability value calculation submodule, which is used to identify the actual needs of the instance application scenario, and in a high-accuracy scenario, the minimum value is used as the probability value of whether the advertising image contains the target item, and in a high-recall scenario, the maximum value is used as the probability value of whether the advertising image contains the target item; a discrimination submodule, which is used to compare the probability value with a preset threshold value, and when the probability value is greater than the preset threshold value, it is determined that the advertising image contains the target item, otherwise it is determined that it does not contain the target item.

[0133] In a preferred embodiment, the target detection model is a Mask-RCNN model, the image recognition model is a ViT model, the basic network architecture of the target detection model is a Mask-RCNN model, the basic network architecture of the image recognition model is a ViT model, and the target object is a flag.

[0134] In order to solve the above technical problems, the embodiment of the present application also provides a computer device. Figure 6 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions, and the database may store a control information sequence, and when the computer-readable instructions are executed by the processor, the processor may implement a target detection method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute the target detection method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0135] In this embodiment, the processor is used to execute Figure 5 The memory stores the program code and various data required to execute the specific functions of each module and its submodule in the target detection device of the present application. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the target detection device of the present application, and the server can call the server's program code and data to execute the functions of all submodules.

[0136] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the target detection method of any embodiment of the present application.

[0137] The present application also provides a computer program product, comprising a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0138] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0139] In summary, this application uses a target detection model to perform target detection on advertising images, obtains a target area image, then performs image recognition on the advertising image and the target area image respectively to obtain a target recognition sequence. Different probability values ​​are then selected based on different application scenarios as the final probability estimate of whether the advertising image contains the target item. This greatly improves the accuracy of target item recognition, helps e-commerce platforms more effectively exclude advertising images that violate relevant regulations, and significantly reduces time and labor costs.

[0140] Those skilled in the art will appreciate that the steps, measures, and schemes in the various operations, methods, and processes discussed in this application may be interchanged, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted.

[0141] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A target detection method, characterized in that: The steps include: Get the advertisement image to be detected; Using a target detection model that has been trained to convergence to perform target detection on the advertisement image to obtain a target area image; Using an image recognition model that has been trained to convergence to perform image recognition on the advertisement image and the target area image respectively, to obtain multiple recognition results, forming a target recognition sequence; The target recognition sequence is used to output the corresponding target recognition result according to the preset conditions of a specific instance scene, including the following steps: Sorting the target recognition sequence to obtain its maximum and minimum values; Identify the actual needs of the application scenario of the example, use the minimum value as the probability value of whether the advertisement image contains the target item in the high-accuracy scenario, and use the maximum value as the probability value of whether the advertisement image contains the target item in the high-recall scenario; The probability value is compared with a preset threshold value, and when the probability value is greater than the preset threshold value, it is determined that the advertisement image contains the target item, otherwise it is determined that the advertisement image does not contain the target item.

2. The target detection method according to claim 1, characterized in that: Obtaining the advertisement image to be detected includes the following steps: Responding to an advertisement publishing request triggered by a user, obtaining the corresponding advertisement publishing information submitted by the user, wherein the advertisement publishing information includes an advertisement picture; The advertisement picture is obtained from the advertisement publishing information.

3. The target detection method according to claim 1, characterized in that: Using a target detection model that has been trained to convergence to perform target detection on the advertisement image to obtain a target area image, the process includes the following steps: A convolutional backbone network is used to extract features from the advertisement image to obtain multi-layer feature maps of different scales; A region generation network is used to generate a plurality of candidate regions of interest for the multi-layer feature maps of different scales, and then an alignment operation of the regions of interest is performed; The head network is used to perform three branch processing on the feature map after alignment of the region of interest, namely bounding box regression processing, recognition processing and mask map prediction, to obtain the detection result; According to the detection result, the target area in the advertisement picture is captured to obtain a corresponding target area image.

4. The target detection method according to claim 1, characterized in that: The image recognition model that has been trained to convergence is used to perform image recognition on the advertisement image and the target area image respectively, and a plurality of recognition results are obtained to form a target recognition sequence, including the following steps: Perform image block embedding on the input image to obtain multiple image block vectors, and add a classification vector to form multiple embedding vectors; Adding a position encoding vector to the multiple embedding vectors to form an input vector, wherein the position encoding vector can maintain spatial position information between image blocks; Stacking multiple encoding modules for the input vector to extract features, wherein the encoding modules include multi-head attention and multi-layer perceptron; A multi-layer perceptron is used to further transform the classification space of the final deep classification vector to obtain its classification probability; The above steps are performed respectively on the advertisement picture and the target area image to obtain a corresponding plurality of recognition results, forming a target recognition sequence of the advertisement picture.

5. The target detection method according to any one of claims 1 to 4, characterized in that: The basic network architecture of the target detection model is the Mask-RCNN model, the basic network architecture of the image recognition model is the ViT model, and the target object is a flag.

6. A target detection device, characterized in that: include: An image acquisition module is used to acquire the advertisement image to be detected; A target detection module is configured to perform target detection on the advertisement image using a target detection model that has been trained to convergence, so as to obtain a target area image; An image recognition module is configured to use an image recognition model that has been trained to convergence to perform image recognition on the advertisement image and the target area image respectively, to obtain a plurality of recognition results, and to form a target recognition sequence; A target identification module, used to output a corresponding target identification result according to preset conditions of a specific instance scene using the target identification sequence; The target identification module comprises: A sorting submodule is used to sort the target recognition sequence to obtain its maximum and minimum values; The probability value calculation submodule is used to identify the actual needs of the instance application scenario, and in a high-accuracy scenario, the minimum value is used as the probability value of whether the advertisement image contains the target item, and in a high-recall scenario, the maximum value is used as the probability value of whether the advertisement image contains the target item; The discrimination submodule is used to compare the probability value with a preset threshold value, and when the probability value is greater than the preset threshold value, it is determined that the advertisement image contains the target object, otherwise it is determined that the advertisement image does not contain the target object.

7. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 5 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images