Image object detection method and device

Abstract

The present application provides a method and device for detecting an object in an image, and relates to the field of Internet of Vehicles; the method comprises: performing density map prediction processing on an image to obtain a density map of the image, wherein the density map represents the density distribution of the object in the image; performing identification clustering processing on the density map to obtain multiple clustered target areas in the density map; performing cropping processing on the image based on the multiple clustered target areas to obtain multiple sub-images corresponding to each other, wherein each sub-image includes multiple objects; performing target detection processing on each sub-image and the image in turn to obtain a first target detection result corresponding to each sub-image and the image; performing fusion processing on each first target detection result to obtain a second target detection result corresponding to the image. Through the present application, various targets of different scales in multimedia images can be effectively detected, and the accuracy of target detection can be improved.

Description

Technical Field

[0001] The present application relates to the fields of Internet of Vehicles and artificial intelligence technology, and in particular to an image target detection method, device, equipment, computer-readable storage medium and computer program product. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0003] Object detection is an important application of artificial intelligence in image processing, and is used in smart transportation, medical care, and other fields.

[0004] In the related art, since the images to be processed are highly random and the image scenes are complex and changeable, the targets are relatively concentrated in some scenes and relatively scattered in some scenes, resulting in low accuracy of target detection. There is no effective solution in the related art on how to effectively detect targets of various scales in images and improve the accuracy of target detection. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium, and computer program product for detecting objects in an image, which can effectively detect objects of various scales in an image and improve the accuracy of object detection.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application provides an image object detection method, including:

[0008] Performing density map prediction processing on the image to obtain a density map of the image, wherein the density map represents the density distribution of the target in the image, and the density map prediction processing includes pooling convolution processing of multiple different scales;

[0009] Performing identification and clustering processing on the density map to obtain a plurality of clustering target areas in the density map;

[0010] Performing cropping processing on the image based on the multiple clustered target areas to obtain multiple sub-images corresponding to each other, wherein each sub-image includes multiple targets;

[0011] Performing target detection processing on each of the sub-images and the image in sequence to obtain a first target detection result corresponding to each of the sub-images and the image;

[0012] Each of the first target detection results is fused to obtain a second target detection result corresponding to the image.

[0013] In the above scheme, both the first pyramid network and the second pyramid network include multiple convolutional neural networks arranged in order of size; the calling of the first pyramid network for the first target detection processing based on the i-th hierarchical processing result of the first backbone subnetwork to obtain the first detection result corresponding to the i-th hierarchical processing result of the first backbone subnetwork includes: calling the N-th convolutional neural network of the first pyramid network for recognition processing based on the N-th hierarchical processing result of the first backbone subnetwork to obtain the N-th recognition result of the first pyramid network; summing the i-th hierarchical processing result of the first backbone subnetwork and the i+1-th recognition result of the first pyramid network to obtain the i+1-th recognition sum processing result; calling the i-th convolutional neural network of the first pyramid network for recognition processing based on the i+1-th recognition sum processing result to obtain the i-th recognition result of the first pyramid network, wherein the size of the i-th convolutional neural network is smaller than the size of the i+1-th convolutional neural network; determining the i-th recognition result of the first pyramid network as the first detection result corresponding to the i-th hierarchical processing result of the first backbone subnetwork.

[0014] The present application provides an image object detection device, including:

[0015] A density map prediction module, used to perform density map prediction processing on an image to obtain a density map of the image, wherein the density map represents the density distribution of the target in the image, and the density map prediction processing includes pooling convolution processing of multiple different scales;

[0016] An identification and clustering module is used to perform identification and clustering processing on the density map to obtain multiple clustering target areas in the density map;

[0017] A cropping module, used for cropping the image based on the multiple clustered target areas to obtain multiple sub-images in one-to-one correspondence, wherein each sub-image includes multiple targets;

[0018] A target detection module, used to perform target detection processing on each of the sub-images and the image in sequence, to obtain a first target detection result corresponding to each of the sub-images and the image in a one-to-one manner;

[0019] A fusion module is used to fuse each of the first target detection results to obtain a second target detection result corresponding to the image.

[0020] An embodiment of the present application provides an electronic device, including:

[0021] A memory for storing executable instructions;

[0022] The processor is used to implement the image target detection method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0023] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute instructions to implement the image target detection method provided by the embodiment of the present application.

[0024] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned method for detecting an object in an image in the embodiment of the present application.

[0025] The embodiments of the present application have the following beneficial effects:

[0026] By clustering the target area obtained by clustering the density map, a sub-image is cropped from the image and identified, so that different parts of the original image are identified separately, so that small-scale targets in the image can be identified more comprehensively, and the detection result of the small-scale target (i.e., the first target detection result) is used as a supplement to the overall target detection result of the image (i.e., the second target detection result), thereby taking into account the detection of large and small targets in the image, effectively detecting targets of various scales in the image, and improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a structural diagram of the target detection system architecture of the image provided by the embodiment of the present application;

[0028] Figure 2 is a schematic diagram of the structure of an image target detection device provided in an embodiment of the present application;

[0029] FIG. 3A to FIG. 3F is a flow chart of an image target detection method provided in an embodiment of the present application;

[0030] 4A to 4H Schematic diagram of the principle of the method for detecting an object in an image provided by an embodiment of the present application;

[0031] FIG. 5A to FIG. 5B It is a schematic diagram of the effect of the target detection method of an image provided by an embodiment of the present application;

[0032] FIG. 5C to FIG. 5E It is a schematic diagram of the principle of the target detection method of the image provided in the embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0034] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0037] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0038] 1) Density map: An image that represents the density distribution of objects in an image. The density points in the density map represent the objects in the image.

[0039] 2) Target: various objective objects in the image. For example, in an aerial image, the target can be a vehicle, building, tree, street light, etc.

[0040] 3) Target detection: also known as target extraction, is an image segmentation technology based on the geometric and statistical characteristics of the target. It combines the segmentation and recognition of the target into one, and its accuracy and real-time performance are important capabilities of the entire system.

[0041] 4) Small-scale objects: objects whose absolute size in the image is smaller than the absolute size threshold, where the absolute size threshold is a set value, for example, the absolute size threshold can be 32×32 mm, or objects whose aspect ratio in the image is lower than the ratio threshold of the original image, for example, the ratio threshold can be

[0042] 5) Large-scale objects: objects whose absolute size in the image is larger than the absolute size threshold. The absolute size threshold is a set value. For example, the absolute size threshold can be 32×32 mm, or objects whose aspect ratio in the image is higher than the ratio threshold of the original image. For example, the ratio threshold can be

[0043] 6) Density distribution: characterizes the density distribution of density points in the density image. The density distribution includes uniform density distribution and non-uniform density distribution. The uniform density distribution characterizes that the density points in the density image are uniformly distributed in the density image, and the non-uniform density distribution characterizes that the density points in the density image are non-uniformly distributed in the density image.

[0044] During the implementation of the embodiments of the present application, the applicant discovered that the related technology has the following problems:

[0045] See also Figure 5A , Figure 5A It is a schematic diagram of the effect of the target detection method of the image provided in the embodiment of the present application. Figure 5A In the images 1, 2 and 3 shown, various targets such as vehicles 13, buildings 11, trees 12, street lights 14 are intricate, and the images collected at different times and locations are different. The scenes are complex and changeable, and the targets in the images (for example, vehicles, buildings, trees, street lights, etc.) are distributed differently. Some targets are more concentrated, some targets are more dispersed, and the distribution differences between different targets are huge.

[0046] In the related technologies of target detection, the detection results are usually directly output in an end-to-end manner. For target detection in high-resolution images, the related technologies usually process the high-resolution images directly. Since high-resolution images occupy a large amount of memory, the processing is time-consuming and memory-intensive, and the hardware requirements are high. Moreover, in the process of target detection, the related technologies perform target detection on high-resolution images, which will cause the loss of small-scale targets, and thus fail to accurately detect small targets in high-resolution images.

[0047] Because in the related art, usually only single scenes can be detected, which cannot meet the needs of complex and changeable scenes, and in the related art, usually only targets of a single scale can be detected, and targets of different scales, especially small-scale targets, cannot be accurately detected. The target detection method of the image provided by the embodiment of the present application can adapt to target detection in different complex scenes, and for target detection of high-resolution images, the processing time is relatively short, and the memory consumption is relatively small, and the hardware requirements are low. Moreover, the detection of large and small targets in the image is taken into account, so as to accurately detect targets of different scales in the image.

[0048] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for detecting targets in an image, which can effectively detect targets of various scales in an image and improve the accuracy of target detection. The following describes exemplary applications of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers.

[0049] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the target detection system 100 for images provided in an embodiment of the present application. To realize application scenarios of target detection (for example, in medical application scenarios, in minimally invasive surgery, target detection is performed on surgical images acquired in real time by acquisition equipment to facilitate accurate determination of lesion sites; in smart transportation application scenarios, target detection is performed on aerial images acquired in real time by aerial photography equipment to facilitate determination of current traffic congestion conditions, thereby performing accurate navigation), a terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0050] The terminal 400 is used for the user to use the client 410, which is displayed on a graphical interface 410-1 (graphic interface 410-1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0051] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, a car terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0052] In some embodiments, the client of terminal 400 receives the image and sends it to server 200 through network 300. Server 200 determines the target detection result corresponding to the image and sends the target detection result to the graphical interface 410-1 in terminal 400 for display.

[0053] In other embodiments, the client of the terminal 400 receives the image, determines the target detection result corresponding to the image, and displays the target detection result in the graphical interface 410 - 1 in the terminal 400 .

[0054] In other embodiments, the terminal 400 (such as a vehicle-mounted terminal) runs a navigation application APP, determines a target detection result of an image in the navigation application APP, and displays the target detection result in a graphical interface 410 - 1 in the terminal 400 .

[0055] See also Figure 2 , Figure 2 is a structural diagram of a server 200 of the image object detection method provided in an embodiment of the present application, Figure 2 The server 200 shown includes: at least one processor 210, a memory 250, and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0056] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0058] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0059] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0060] The operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0061] The network communication module 252 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include: Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB).

[0062] In some embodiments, the image object detection device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 The target detection device 255 of the image stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a density map prediction module 2551, a recognition clustering module 2552, a cropping module 2553, a target detection module 2554, and a fusion module 2555. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0063] The target detection method of the image provided by the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided by the embodiment of the present application.

[0064] In some embodiments, see Figure 4A , Figure 4AIt is a schematic diagram of the principle of the target detection method for an image provided by an embodiment of the present application. The density map prediction network is used to predict the density map of an image to obtain a density map of the image. The target detection network includes a backbone network, a pyramid network and a cascade network. The target detection network is used to perform target detection on images and sub-images. Based on the image, the density map prediction network is called to perform density map prediction processing to obtain a density map of the image; based on the density map of the image, multiple sub-images of the image are obtained. Based on the image and multiple sub-images, the target detection network is called to perform target detection processing to obtain a detection result. Thereby, targets of various scales in the image are effectively detected, and the accuracy of target detection is improved.

[0065] See also Figure 3A , Figure 3A is a flow chart of the target detection method for an image provided by an embodiment of the present application, which will be combined with Figure 3A Steps 101 to 105 are shown for illustration. The execution subject of the following steps 101 to 105 can be the aforementioned server or terminal.

[0066] In step 101, a density map prediction process is performed on an image to obtain a density map of the image, wherein the density map represents the density distribution of objects in the image.

[0067] In some embodiments, the target in the image refers to various objective objects existing in the image. For example, the target may be an objective object such as a vehicle, a building, a tree, a street lamp, etc. in the image.

[0068] As an example, see Figure 5D , Figure 5D Schematic diagram of the principle of the target detection method for an image provided in the embodiment of the present application. When the image is an aerial image, the aerial image is subjected to density map prediction processing to obtain a density map of the image, such as Figure 5D The density map shown represents the density distribution of targets in the aerial image, and the density map prediction process includes pooling convolution processes of multiple scales.

[0069] In some embodiments, performing density map prediction processing on the image in the above step 101 to obtain the density map of the image can be achieved in the following way: based on the image, calling a trained density map prediction network to perform density map prediction processing to obtain the density map of the image.

[0070] Among them, the trained density map prediction network is trained in the following way: based on the image samples, the density map prediction network to be trained is called to perform density map prediction processing to obtain the predicted density map of the image samples; based on the predicted density map of the image samples and the true density map of the image samples, the density loss function is called to obtain the loss value of the image samples; based on the loss value of the image samples, the parameters of the density map prediction network to be trained are updated.

[0071] As an example, the density loss function may be Euclidean distance, Hamming distance, etc. When the density loss function is Euclidean distance, the expression of the density loss function may be:

[0072]

[0073] Among them, L(θ) represents the density loss function, F(X i ,θ) represents the predicted density map of the image sample, F i Represents the true density map of the image sample, and N represents the number of calls to the density map prediction network.

[0074] In some embodiments, see Figure 3B , Figure 3B is a flow chart of an image target detection method provided in an embodiment of the present application, Figure 3B The illustrated step 101 can be implemented by calling a trained density map prediction network based on an image to execute steps 1011 to 1013, which are described below respectively.

[0075] In step 1011, the image is subjected to a plurality of pooling convolution processes of different scales to obtain a plurality of convolution feature maps of different scales.

[0076] In some embodiments, the convolution and pooling process may include a convolution process and a pooling process, the scale of the convolution process refers to the size of the convolution kernel of the convolution process, and the scale of the pooling process refers to the step size of the pooling process. The convolution process is used to determine the feature map of the image, and the pooling process is used to reduce the dimension of the image.

[0077] As an example, see Figure 4D , image 1 is convolved at three scales (i.e., convolution 9×9, convolution 7×7, and convolution 5×5, where 9×9 represents the size of the convolution kernel) to obtain feature map 16 (16 represents the number of feature maps), feature map 20 (20 represents the number of feature maps), and feature map 24 (24 represents the number of feature maps). Feature map 16 is sequentially pooled (pooled 2×2) and convolved (convolved 7×7) to obtain feature map 32 (32 represents the number of feature maps), and feature map 32 is sequentially pooled (pooled 2×2) and convolved (convolved 7×7) to obtain feature map 16. Feature map 16 is convolved (convolved 7×7) to obtain convolution feature map 8. Similarly, feature map 20 is subjected to multiple pooling processes (pooling 2×2) and convolution processes (convolution 5×5) to obtain convolution feature map 10; feature map 24 is subjected to multiple pooling processes (pooling 2×2) and convolution processes (convolution 3×3) to obtain convolution feature map 12. That is, multiple pooling and convolution processes of different scales are performed on image 1 to obtain convolution feature maps 8, 10, and 12 of different scales.

[0078] In step 1012, a plurality of convolutional feature maps of different scales are merged to obtain a merged feature map.

[0079] In some embodiments, the merging process refers to the process of merging multiple convolutional feature maps into a merged feature map.

[0080] As an example, see Figure 4D , the convolution feature maps 8, 10 and 12 of different scales are merged to obtain a merged feature map 30 (where 30 represents the number of merged feature maps).

[0081] In step 1013, convolution processing is performed on the merged feature map to obtain a density map of the image.

[0082] As an example, see Figure 4D , convolution processing is performed on the combined feature map 30 to obtain the features of the image Figure 2 .

[0083] In this way, by performing density map prediction processing on the image, the density map of the image is obtained, which is convenient for subsequent processing of the density map to crop the image to obtain sub-images, and perform target detection processing on the sub-images, thereby facilitating the subsequent accurate identification of small-scale targets in the image.

[0084] In step 102, identification and clustering processing is performed on the density map to obtain a plurality of clustering target areas in the density map.

[0085] In some embodiments, the cluster target region in the density map represents a region in the density map where the target distribution is relatively dense. A cluster target region includes at least one target. The identification clustering process is used to identify and cluster the density points in the density map.

[0086] In some embodiments, see Figure 3B , Figure 3B The step 102 shown in which identification and clustering processing is performed on the density map can be implemented by executing steps 1021 to 1022, which are described below respectively.

[0087] In step 1021, the density points in the density map are identified to obtain the probability distribution of the density points.

[0088] In some embodiments, the identification of density points in the density map can be achieved through a Gaussian mixture model. Specifically, in probability statistics, the probability density function of the multi-dimensional single Gaussian model can be:

[0089]

[0090] Among them, P(x|θ) represents the probability density function, μ represents the expectation of the multidimensional single Gaussian model, |∑| represents the variance of the multidimensional single Gaussian model, and x represents the density map.

[0091] When the Gaussian mixture model includes K multi-dimensional single Gaussian models, the probability distribution function of the Gaussian mixture model can be expressed as:

[0092]

[0093] Among them, K represents the number of multidimensional single Gaussian models that make up the multidimensional Gaussian mixture model, k represents the label of the multidimensional single Gaussian model, and α k The weighted coefficients representing the multidimensional single Gaussian model labeled k, Characterize the probability density function of the multidimensional single Gaussian model labeled k.

[0094] As an example, by identifying the density points in the density map, the probability density function of the multidimensional single Gaussian model to which each density point in the density map obeys can be determined. For example, density point A in the density map obeys the probability density function of the multidimensional single Gaussian model A, and density point B in the density map obeys the probability density function of the multidimensional single Gaussian model B, that is, the density points in the density map obey K multidimensional single Gaussian models respectively.

[0095] In step 1022, density points that obey the same probability density function are clustered according to the probability distribution of the density points to obtain a plurality of cluster target areas in the density map.

[0096] As an example, density points that obey the same probability density function are clustered according to their probability distribution. When the Gaussian mixture model includes K multidimensional single Gaussian models, the density points in the density map can be clustered into K categories, thereby obtaining K clustering target areas in the density map.

[0097] In this way, through the probability distribution of density points in the density map, multiple clustered target areas in the density map are determined, so that the image is initially divided into multiple areas according to the degree of aggregation of the targets in the image, so that the preliminary detection of the targets in the image is achieved through the density map. It is convenient to crop the image later to obtain multiple sub-images of the image, which provides a good data foundation for subsequent higher-precision target detection.

[0098] In step 103, the image is cropped based on the multiple clustered target regions to obtain multiple sub-images in one-to-one correspondence, wherein each sub-image includes multiple targets.

[0099] In some embodiments, since the size of the density map is consistent with the size of the image, the above-mentioned step 103 performs cropping on the image based on multiple clustering target areas to obtain multiple sub-images corresponding to each other, which can be achieved in the following way: perform the following processing on any clustering target area: obtain the size of the sub-image and the position of the sub-image in the image based on the size of each clustering target area and the position of the clustering target area in the density map; and crop the image based on the size of the sub-image and the position of the sub-image in the image to obtain the sub-image.

[0100] As an example, see Figure 4B , Figure 4B 4 is a schematic diagram of the principle of the target detection method for an image provided by an embodiment of the present application. According to the size of the clustered target region 41 and the position of the clustered target region 41 in the density map, the size of the sub-image 42 and the position of the sub-image 42 in the image 43 are obtained; based on the size of the sub-image 42 and the position of the sub-image 42 in the image 43, the image 43 is cropped to obtain the sub-image 42.

[0101] In step 104, target detection processing is performed on each sub-image and image in turn to obtain a first target detection result corresponding to each sub-image and image.

[0102] As an example, see Figure 4E After performing multi-scale feature extraction processing, first target detection processing, and second target detection processing on each sub-image and image in turn, a first target detection result corresponding to each sub-image and image is obtained.

[0103] In some embodiments, see Figure 3C , Figure 3C is a flow chart of an image target detection method provided in an embodiment of the present application, Figure 3C The illustrated step 104 may perform the following steps 1041 to 1044 for any sub-image, which are described below respectively.

[0104] In step 1041, a multi-scale feature extraction process is performed on the sub-image to obtain features of multiple different scales corresponding to the sub-image.

[0105] In some embodiments, the multi-scale feature extraction process is implemented by a backbone network, which includes a convolutional network, at least one first backbone sub-network and at least one second backbone sub-network of the same structure; the first backbone sub-network and the second backbone sub-network both include multiple cascaded hierarchical networks, and the sizes of the multiple cascaded hierarchical networks are different. The backbone network is used to extract features of multiple different scales from the image. The first backbone sub-network is used to enhance the feature expression capability of the second backbone sub-network, wherein the relationship expression between the first backbone sub-network and the second backbone sub-network can be:

[0106]

[0107] Among them, g l-1 Represents the fusion link, the output features of the first backbone sub-network are passed through g l-1 After conversion, it serves as the input of each layer of the second backbone sub-network. Represents the output of the lth level network in the second backbone sub-network, Represents the output of the l-1th level network in the second backbone sub-network.

[0108] As an example, see Figure 4F , Figure 4F Schematic diagram of the principle of the method for detecting an object in an image provided by the embodiment of the present application. The backbone network 12 includes a convolutional network (see Figure 4F , convolution 41), at least one first backbone sub-network 121 and at least one second backbone sub-network 122 with the same structure, the number of first backbone sub-networks 121 and the number of second backbone sub-networks 122 are the same; the first backbone sub-networks 121 and the second backbone sub-networks 122 each include a plurality of cascaded hierarchical networks, the first backbone sub-network 121 includes four cascaded hierarchical networks (hierarchical network 42, hierarchical network 43, hierarchical network 44, hierarchical network 45), and the size of each cascaded hierarchical network is different.

[0109] In some embodiments, see Figure 3D , Figure 3D is a flow chart of an image target detection method provided in an embodiment of the present application, Figure 3D The step 1041 shown performs multi-scale feature extraction processing on the sub-image, which can be performed by executing steps 10411 to 10414, which are described below respectively.

[0110] In step 10411, the convolution network is called to perform convolution processing based on the sub-image to obtain the convolution processing result of the sub-image.

[0111] As an example, see Figure 4FWhen the input is a sub-image, the convolution network 41 is called based on the sub-image to perform convolution processing to obtain the convolution processing result of the sub-image. The convolution network is used to perform convolution processing on the input to obtain the convolution processing result.

[0112] In step 10412, multiple cascaded hierarchical networks in the first backbone sub-network are called to perform hierarchical processing based on the convolution processing result to obtain the hierarchical processing result of each hierarchical network in the first backbone sub-network.

[0113] As an example, see Figure 4F Based on the convolution processing result, the four cascaded hierarchical networks (hierarchical network 42, hierarchical network 43, hierarchical network 44, hierarchical network 45) in the first backbone sub-network 121 are called to perform hierarchical processing to obtain the hierarchical processing results of each hierarchical network in the first backbone sub-network, that is, the hierarchical processing results of hierarchical network 42, the hierarchical processing results of hierarchical network 43, the hierarchical processing results of hierarchical network 44, and the hierarchical processing results of hierarchical network 45.

[0114] In some embodiments, the above step 10412 can be implemented in the following manner: based on the convolution processing result, calling the 1st level network in the first backbone sub-network for hierarchical processing, and obtaining the 1st hierarchical processing result of the first backbone sub-network; iterating i to perform the following processing: based on the i-th hierarchical processing result of the first backbone sub-network, calling the i+1th level network in the first backbone sub-network for hierarchical processing, and obtaining the i+1th hierarchical processing result of the first backbone sub-network; wherein, i is an integer variable with increasing values ​​and the value satisfies 1≤i≤N-1, N is the total number of multiple cascaded hierarchical networks, and the size of the i-th level network is larger than the size of the i+1th level network.

[0115] As an example, see Figure 4F , based on the convolution processing result, the first hierarchical network 42 in the first backbone sub-network 121 is called for hierarchical processing to obtain the first hierarchical processing result of the first backbone sub-network 121; when i=1, based on the first hierarchical processing result of the first backbone sub-network 121, the second hierarchical network 43 in the first backbone sub-network 121 is called for hierarchical processing to obtain the second hierarchical processing result of the first backbone sub-network 121. When i=2, based on the second hierarchical processing result of the first backbone sub-network 121, the third hierarchical network 44 in the first backbone sub-network 121 is called for hierarchical processing to obtain the third hierarchical processing result of the first backbone sub-network. When the total number of hierarchical networks is 4, that is, N=4, 1≤i≤3, the size of the first hierarchical network 42 is larger than the size of the second hierarchical network 43, and the size of the second hierarchical network 43 is larger than the size of the third hierarchical network 44.

[0116] In some embodiments, the hierarchical network includes a regular hierarchical network and a shifted hierarchical network. The above-mentioned calling the i+1th hierarchical network in the first backbone subnetwork for hierarchical processing based on the i-th hierarchical processing result of the first backbone subnetwork to obtain the i+1th hierarchical processing result of the first backbone subnetwork can be achieved in the following manner: calling the i+1th regular hierarchical network in the first backbone subnetwork for hierarchical processing based on the i-th hierarchical processing result of the first backbone subnetwork to obtain the i+1th regular hierarchical processing result of the first backbone subnetwork; calling the i+1th shifted hierarchical network in the first backbone subnetwork for hierarchical processing based on the i+1th regular hierarchical processing result of the first backbone subnetwork, and determining the i+1th shifted hierarchical processing result of the first backbone subnetwork as the i+1th hierarchical processing result of the first backbone subnetwork.

[0117] As an example, see Figure 4F , the hierarchical network 11 includes a regular hierarchical network 47 and a shifted hierarchical network 48. When i=1, based on the first hierarchical processing result of the first backbone sub-network 121, the second regular hierarchical network 47 in the first backbone sub-network 121 is called for hierarchical processing to obtain the second regular hierarchical processing result of the first backbone sub-network 121; based on the second regular hierarchical processing result of the first backbone sub-network 121, the second shifted hierarchical network 48 in the first backbone sub-network 121 is called for hierarchical processing to obtain the second shifted hierarchical processing result of the first backbone sub-network 121, and the obtained second hierarchical processing result of the first backbone sub-network 121 is determined as the second hierarchical processing result of the first backbone sub-network 121.

[0118] In some embodiments, the rule hierarchical network includes a rule window, a normalization processing layer and a multi-layer perception layer; the above-mentioned calling the i+1th rule hierarchical network in the first backbone subnetwork for hierarchical processing based on the i-th hierarchical processing result of the first backbone subnetwork to obtain the i+1th rule hierarchical processing result of the first backbone subnetwork can be implemented in the following manner: calling the normalization processing layer for normalization processing based on the i-th hierarchical processing result of the first backbone subnetwork to obtain a first normalization processing result; calling the rule window for self-attention processing based on the normalization processing result to obtain the rule window processing result; summing the i-th hierarchical processing result of the first backbone subnetwork and the rule window processing result to obtain the rule window sum processing result; calling the normalization processing layer for normalization processing based on the rule window sum processing result to obtain a second normalization processing result; calling the multi-layer perception layer for recognition processing based on the second normalization processing result to obtain a first recognition processing result; summing the first recognition processing result and the rule window sum processing result, and determining the obtained first perception layer sum processing result as the i+1th rule hierarchical processing result of the first backbone subnetwork.

[0119] In some embodiments, the normalization layer (Layer Normalizaiton, LN) is used to normalize the input of the normalization layer. The normalization process can ensure the consistency of the sample data distribution by normalizing all the neuron nodes of a single sample. The rule window (Windows Multi-head Self Attention, W-MSA) is used to perform self-attention processing (Self-Attention) in the rule window, wherein the rule window evenly divides the image in a non-overlapping and non-shifting manner, and the self-attention process is used to focus on some detailed features of the normalization processing results. The multi-layer perceptron (Multilayer perceptron, MLP) includes an input layer, an output layer and a hidden layer, wherein the input layer and the hidden layer, and the hidden layer and the output layer are fully connected, and the multi-layer perceptron is used to perform recognition processing on the second normalization processing result, specifically by fitting the recognition function through the hidden layer in the multi-layer perceptron, so that the multi-layer perceptron has a recognition function.

[0120] As an example, see Figure 4F , the rule hierarchical network 47 includes a rule window, a normalization processing layer and a multi-layer perception layer. When i=1, the normalization processing layer is called to perform normalization processing based on the first hierarchical processing result of the first backbone sub-network 121 to obtain a first normalization processing result; the rule window is called to perform self-attention processing based on the normalization processing result to obtain a rule window processing result; the first hierarchical processing result of the first backbone sub-network and the rule window processing result are summed to obtain a rule window summed processing result; the normalization processing layer is called to perform normalization processing based on the rule window summed processing result to obtain a second normalized processing result; the multi-layer perception layer is called to perform recognition processing based on the second normalized processing result to obtain a first recognition processing result; the first recognition processing result and the rule window summed processing result are summed, and the obtained first perception layer summed processing result is determined as the second rule hierarchical processing result of the first backbone sub-network.

[0121] As an example, the expression of the rule window processing result can be:

[0122]

[0123] Among them, W-MSA represents the process of self-attention processing in the regular window, LN represents the process of normalization processing in the normalization processing layer, and z l-1 Represents the i-th hierarchical processing result, Characterizes the rule window summation processing result.

[0124] As an example, the expression of the sum processing result of the first perception layer can be:

[0125]

[0126] in, It represents the summing result of the first perception layer, MLP represents the recognition process of the multi-layer perception layer, and LN represents the normalization process of the normalization layer. Characterizes the rule window summation processing result.

[0127] In some embodiments, the shifted hierarchical network includes a shift window, a normalization processing layer and a multi-layer perception layer; the above-mentioned calling the i+1th shifted hierarchical network in the first backbone subnetwork for hierarchical processing based on the i+1th rule hierarchical processing result of the first backbone subnetwork to obtain the i+1th shifted hierarchical processing result of the first backbone subnetwork can be achieved in the following manner: calling the normalization processing layer for normalization processing based on the i+1th rule hierarchical processing result of the first backbone subnetwork to obtain a third normalization processing result; calling the shifted window for self-attention processing based on the third normalization processing result to obtain the shifted hierarchical network. The method comprises the following steps: obtaining an i+1th regular hierarchical processing result and a shift window processing result; summing up the i+1th regular hierarchical processing result and the shift window processing result of the first backbone sub-network to obtain a shift window sum processing result; calling a normalization processing layer based on the shift window sum processing result to perform normalization processing to obtain a fourth normalization processing result; calling a multi-layer perception layer based on the fourth normalization processing result to perform recognition processing to obtain a second recognition processing result; summing up the second recognition processing result and the shift window sum processing result, and determining the obtained second perception layer sum processing result as the i+1th shift hierarchical processing result of the first backbone sub-network.

[0128] In some embodiments, a shift window (Windows Multi-head Self Attention, SW-MSA) is used to perform self-attention processing (Self-Attention) in the shift window, wherein the shift window evenly divides the image in an overlapping and shifted manner, and the self-attention processing is used to focus on some detail features of the normalized processing results.

[0129] As an example, see Figure 4F, the shifted hierarchical network 48 includes a shifted window, a normalization processing layer and a multi-layer perception layer. When i=1, the normalization processing layer is called to perform normalization processing based on the second rule hierarchical processing result of the first backbone sub-network 121 to obtain a third normalization processing result; the shifted window is called to perform self-attention processing based on the third normalization processing result to obtain a shifted window processing result; the second rule hierarchical processing result of the first backbone sub-network and the shifted window processing result are summed to obtain a shifted window summed processing result; the normalization processing layer is called to perform normalization processing based on the shifted window summed processing result to obtain a fourth normalized processing result; the multi-layer perception layer is called to perform recognition processing based on the fourth normalized processing result to obtain a second recognition processing result; the second recognition processing result and the shifted window summed processing result are summed, and the obtained second perception layer summed processing result is determined as the second shifted hierarchical processing result of the first backbone sub-network 121.

[0130] As an example, the expression of the shift window sum processing result can be:

[0131]

[0132] in, represents the result of the shift window summing process, SW-MSA represents the process of self-attention processing of the shift window, and LN represents the process of normalization processing of the normalization processing layer. Characterizes the i+1th rule hierarchical processing result of the first backbone sub-network.

[0133] As an example, the expression of the i+1th shift hierarchical processing result of the first backbone sub-network can be:

[0134]

[0135] in, represents the i+1th shift hierarchical processing result of the first backbone subnetwork, MLP represents the recognition processing process of the multi-layer perception layer, LN represents the normalization processing process of the normalization processing layer, Characterize the shift window summation processing result.

[0136] In step 10413, based on the convolution processing result and the hierarchical processing result of each hierarchical network in the first backbone sub-network, multiple hierarchical networks in the second backbone sub-network are called to perform hierarchical processing to obtain the hierarchical processing result of each hierarchical network in the second backbone sub-network.

[0137] As an example, see Figure 4FBased on the convolution processing results and the hierarchical processing results of each hierarchical network in the first backbone sub-network 121, multiple hierarchical networks (hierarchical network 46, hierarchical network 47, hierarchical network 48, hierarchical network 49) in the second backbone sub-network 122 are called for hierarchical processing to obtain the hierarchical processing results of each hierarchical network in the second backbone sub-network 122.

[0138] In some embodiments, the above step 10413 can be implemented in the following manner: add the convolution processing result and the hierarchical processing result of the i-th hierarchical network of the first backbone subnetwork to obtain the first summed processing result; based on the first summed processing result, call the first hierarchical network in the second backbone subnetwork for hierarchical processing to obtain the first hierarchical processing result of the second backbone subnetwork; iterate i and perform the following processing: add the i-th hierarchical processing result of the first backbone subnetwork and the i-th hierarchical processing result of the second backbone subnetwork to obtain the i+1th summed processing result, call the i+1th hierarchical network in the second backbone subnetwork for hierarchical processing based on the i+1th summed processing result to obtain the i+1th hierarchical processing result of the second backbone subnetwork.

[0139] As an example, see Figure 4F , add the convolution processing result, the hierarchical processing result of the 4th level network 45 of the first backbone sub-network 121, the hierarchical processing result of the 3rd level network 44 of the first backbone sub-network 121, the hierarchical processing result of the 2nd level network 43 of the first backbone sub-network 121 and the hierarchical processing result of the 1st level network 42 of the first backbone sub-network 121 to obtain the first summed processing result; based on the first summed processing result, call the 1st level network 46 in the second backbone sub-network 122 for hierarchical processing to obtain the first hierarchical processing result of the second backbone sub-network 122.

[0140] In step 10414, the hierarchical processing results of each hierarchical network in the first backbone sub-network and the hierarchical processing results of each hierarchical network in the second backbone sub-network are determined as features of multiple different scales corresponding to the sub-image.

[0141] As an example, see Figure 4FThe hierarchical processing results of the first hierarchical network 42 in the first backbone sub-network 121, the hierarchical processing results of the second hierarchical network 43 in the first backbone sub-network 121, the hierarchical processing results of the third hierarchical network 44 in the first backbone sub-network 121, the hierarchical processing results of the fourth hierarchical network 45 in the first backbone sub-network 121, and the hierarchical processing results of the first hierarchical network 46 in the second backbone sub-network 122, the hierarchical processing results of the second hierarchical network 47 in the second backbone sub-network 122, the hierarchical processing results of the third hierarchical network 48 in the second backbone sub-network 122, and the hierarchical processing results of the fourth hierarchical network 49 in the second backbone sub-network 122 are determined as features of multiple different scales corresponding to the sub-image.

[0142] In step 1042, a first target detection process is performed on features of multiple different scales to obtain a first detection result corresponding to the feature of each scale.

[0143] In some embodiments, the first target detection processing is implemented by at least one first pyramid network and at least one second pyramid network of the same structure, the number of first pyramid networks is the same as the number of first backbone sub-networks, and the number of second pyramid networks is the same as the number of second backbone sub-networks.

[0144] As an example, see Figure 4F The first target detection process is implemented by a first pyramid network 131 and a second pyramid network 132 of the same structure. The number of first pyramid networks 131 and the number of first backbone sub-networks 121 are both 1, and the number of second pyramid networks 132 and the number of second backbone sub-networks 122 are both 1.

[0145] In some embodiments, see Figure 3E , Figure 3E is a flow chart of an image target detection method provided in an embodiment of the present application, Figure 3E In step 1042 shown, the first target detection process is performed on features of multiple different scales. This can be achieved by executing the following steps 10421 to 10423 for features of any scale corresponding to the sub-image, which are described below.

[0146] In step 10421, the first pyramid network is called to perform a first target detection process based on the i-th hierarchical processing result of the first backbone sub-network, and a first detection result corresponding to the i-th hierarchical processing result of the first backbone sub-network is obtained.

[0147] In some embodiments, a pyramid network (Feature Pyramid Networks, FPN) is used to perform target detection on the hierarchical processing results of the backbone network to obtain detection results corresponding to the hierarchical processing results of each scale in the backbone network. The pyramid network includes multiple convolutional neural networks arranged in order of size. After convolution processing by convolutional neural networks of different sizes, the size of the feature map gradually becomes smaller, thereby gradually and accurately extracting the target features.

[0148] In some embodiments, both the first pyramid network and the second pyramid network include multiple convolutional neural networks arranged in order of size, and the above step 10421 can be implemented in the following manner: based on the Nth hierarchical processing result of the first backbone subnetwork, the Nth convolutional neural network of the first pyramid network is called for recognition processing to obtain the Nth recognition result of the first pyramid network; the i-th hierarchical processing result of the first backbone subnetwork and the i+1th recognition result of the first pyramid network are summed to obtain the i+1th recognition sum processing result; based on the i+1th recognition sum processing result, the i-th convolutional neural network of the first pyramid network is called for recognition processing to obtain the i-th recognition result of the first pyramid network, wherein the size of the i-th convolutional neural network is smaller than the size of the i+1th convolutional neural network; the i-th recognition result of the first pyramid network is determined as the first detection result corresponding to the i-th hierarchical processing result of the first backbone subnetwork.

[0149] In some embodiments, a convolutional neural network (CNN) is used to perform translation-invariant classification on input information according to its hierarchical structure, perform a first target detection process on features of a scale, and obtain the location and category of the target.

[0150] As an example, see Figure 4FWhen N=4, the fourth convolutional neural network 1314 of the first pyramid network 131 is called for recognition processing based on the fourth hierarchical processing result of the first backbone sub-network 121 (the fourth hierarchical processing result of the fourth hierarchical network 45 of the first backbone sub-network) to obtain the fourth recognition result of the first pyramid network. The third hierarchical processing result of the first backbone sub-network 121 and the fourth recognition result of the first pyramid network 131 are added together to obtain the fourth recognition addition processing result; the third convolutional neural network 1313 of the first pyramid network 131 is called for recognition processing based on the fourth recognition addition processing result to obtain the third recognition result of the first pyramid network 131. The first recognition result of the first pyramid network 131 is determined as the first detection result corresponding to the first hierarchical processing result of the first backbone sub-network 121, and the second recognition result of the first pyramid network 131 is determined as the first detection result corresponding to the second hierarchical processing result of the first backbone sub-network 121, etc.

[0151] In step 10422, the second pyramid network is called to perform a first target detection process based on the i-th hierarchical processing result of the second backbone sub-network, and a first detection result corresponding to the i-th hierarchical processing result of the second backbone sub-network is obtained.

[0152] As an example, see Figure 4F When i=3, the second pyramid network 132 is called to perform the first target detection process based on the third hierarchical processing result of the second backbone sub-network 122 to obtain the first detection result corresponding to the third hierarchical processing result of the second backbone sub-network 122.

[0153] In step 10423, the first detection result corresponding to the i-th hierarchical processing result of the first backbone subnetwork and the first detection result corresponding to the i-th hierarchical processing result of the second backbone subnetwork are fused to obtain the first detection result corresponding to the feature of each scale.

[0154] In some embodiments, the process of fusing the first detection result corresponding to the i-th hierarchical processing result of the first backbone subnetwork and the first detection result corresponding to the i-th hierarchical processing result of the second backbone subnetwork may be a process of summing the first detection result corresponding to the i-th hierarchical processing result of the first backbone subnetwork and the first detection result corresponding to the i-th hierarchical processing result of the second backbone subnetwork. The expression of the summing process may be:

[0155] A=B+C (9)

[0156] Among them, A represents the first detection result corresponding to the feature of each scale, B represents the first detection result corresponding to the i-th hierarchical processing result of the first backbone subnetwork, and C represents the first detection result corresponding to the i-th hierarchical processing result of the second backbone subnetwork.

[0157] As an example, see Figure 4F , the first detection result corresponding to the first hierarchical processing result of the first backbone subnetwork 121 and the first detection result corresponding to the first hierarchical processing result of the second backbone subnetwork 122 are fused to obtain a first detection result corresponding to the feature of the first scale; the first detection result corresponding to the second hierarchical processing result of the first backbone subnetwork 121 and the first detection result corresponding to the second hierarchical processing result of the second backbone subnetwork 122 are fused to obtain a first detection result corresponding to the feature of the second scale.

[0158] In step 1043, a second target detection process is performed on each first detection result to obtain a second detection result corresponding to each first detection result.

[0159] As an example, see Figure 4F , the first detection result corresponding to the feature of the first scale is subjected to a second detection process to obtain a second detection result corresponding to the first detection result corresponding to the feature of the first scale. The first detection result corresponding to the feature of the second scale is subjected to a second detection process to obtain a second detection result corresponding to the first detection result corresponding to the feature of the second scale.

[0160] The detection accuracy of the second target detection process is greater than the detection accuracy of the first target detection process, and the detection accuracy represents the accuracy of the position and category of the target detected by the target detection process.

[0161] In some embodiments, the second target detection process is implemented by a cascade network, which includes a region proposal network and multiple pooling networks. The region proposal network (RPN) is used to generate a detection box, and the pooling network is used to remove redundant information in the region proposal processing results and refine the first target detection results of different scales.

[0162] As an example, see Figure 4F The second target detection process is implemented through a cascade network 14, which includes a region generation network and pooling network 1, pooling network 2, pooling network 3, pooling network 4, etc.

[0163] In some embodiments, see Figure 3F , Figure 3F is a flow chart of an image target detection method provided in an embodiment of the present application, Figure 3F The shown step 1043 can execute the following steps 10431 to 10433 for any first detection result, which are described below respectively.

[0164] In step 10431, the region generation network is called to perform region generation processing based on the first detection result to obtain a region generation processing result corresponding to the first detection result.

[0165] As an example, see Figure 4F The first detection result is the cascade network input. Based on the cascade network input, the region generation network is called to perform region generation processing to obtain the region generation processing result corresponding to the cascade network input. Figure 4F There are 4 cascade network inputs shown in the figure. Then, the region generation network can be called separately based on these 4 cascade network inputs to perform region generation processing.

[0166] In step 10432, multiple pooling networks are called to perform pooling processing based on the region generation processing result and the first detection result to obtain a pooling processing result corresponding to each pooling network.

[0167] As an example, see Figure 4F Based on the region generation processing result and the first detection result (i.e., the cascade network input), pooling network 1, pooling network 2, pooling network 3, and pooling network 4 are called to perform pooling processing, and pooling processing result 1 corresponding to pooling network 1, pooling processing result 2 corresponding to pooling network 2, pooling processing result 3 corresponding to pooling network 3, and pooling processing result 4 corresponding to pooling network 4 are obtained.

[0168] In some embodiments, the above step 10432 can be implemented in the following way: based on the region generation processing result and the first detection result, calling the first pooling network for pooling processing, and obtaining the first pooling processing result corresponding to the first pooling network; based on the jth pooling processing result and the first detection result, calling the j+1th pooling network for pooling processing, and obtaining the j+1th pooling processing result corresponding to the j+1th pooling network, wherein 1≤j≤M-1, M is a positive integer greater than or equal to 2, M is the total number of multiple pooling networks, and the accuracy of the j+1th pooling processing result is greater than that of the jth pooling processing result.

[0169] As an example, see Figure 4F , based on the region generation processing result and the first detection result (cascade network input), the first pooling network 1 is called to perform pooling processing, and the first pooling processing result corresponding to the first pooling network 1 is obtained; based on the first pooling processing result and the first detection result, the second pooling network 2 is called to perform pooling processing, and the second pooling processing result corresponding to the second pooling network 2 is obtained, and the accuracy of the second pooling processing result is greater than that of the first pooling processing result.

[0170] In step 10433, the pooling processing result is determined as the second detection result.

[0171] As an example, see Figure 4F , pooling processing result 1, pooling processing result 2, pooling processing result 3, and pooling processing result 4 can all be determined as the second detection result; or pooling processing result 4 with relatively high accuracy can be determined as the second detection result.

[0172] In step 1044, the second detection result is determined as the first target detection result.

[0173] In this way, multi-scale feature extraction processing is implemented through the backbone network to obtain features of multiple different scales corresponding to the sub-image, and the first target detection processing is implemented through the pyramid network, thereby realizing the first target detection of multiple different scales by divide-and-conquer, that is, the first target detection processing realizes the first target detection of different scales by features of different scales. The second target detection processing is implemented through the cascade network, that is, the first target detection results of different scales are refined multiple times using multiple pooling networks, thereby greatly improving the accuracy of target detection.

[0174] In some embodiments, in the above step 104, target detection processing is performed on each sub-image and image in turn to obtain a first target detection result corresponding to each sub-image and image. Figure 4F The target detection network shown can also be implemented by Figure 4G The generic object detection network implementation shown. Figure 4G , Figure 4G6 is a schematic diagram of the principle of the universal detection network provided in an embodiment of the present application. The universal detection network includes a backbone network model 601, a region recommendation network model 602 and a segmentation network 604. Take the input of the universal detection network as a sub-image as an example for explanation. Based on the sub-image, the backbone network model 601 is called to perform feature extraction processing to obtain a feature map of the sub-image. Based on the feature map of the sub-image, the convolution network is called (the convolution kernel size of the convolution network can be 3×3) for convolution processing to obtain a first feature map. Based on the first feature map, the convolution network is called (the convolution kernel size of the convolution network can be 1×1) for convolution processing to obtain a second feature map. The second feature map is subjected to logistic regression processing and border regression processing respectively, and the logistic regression processing results and the border regression processing results are merged to obtain a recommended area. The feature map is cropped based on the recommended area to obtain a cropped feature map. Based on the cropped feature map, the fully connected layer 1 is called for processing to obtain the processing result of the fully connected layer 1. Based on the processing results of the fully connected layer 1, the fully connected layer 2 and the fully connected layer 3 are called for processing respectively to obtain the processing results of the fully connected layer 2 and the processing results of the fully connected layer 3. The processing results of the fully connected layer 2 are subjected to border regression processing to obtain the position of the detected target. The processing results of the fully connected layer 3 are subjected to logistic regression processing to obtain the category of the detected target. Among them, the above-mentioned logistic regression processing can be implemented by a logistic regression function (softmax function), and the border detection processing can be implemented by a border detection function (bbox reg function).

[0175] In this way, the target detection of sub-images can be achieved through the general detection network, but relatively speaking, Figure 4F The detection accuracy of the object detection network shown is higher than Figure 4G The general detection network shown.

[0176] In step 105, each first target detection result is fused to obtain a second target detection result corresponding to the image.

[0177] In some embodiments, in the above step 105, each first target detection result is fused to obtain a second target detection result corresponding to the image by: fusing the first target detection result corresponding to the image with the first detection result corresponding to each sub-image to obtain the second target detection result corresponding to the image. The fusion process can be implemented by non-maximum suppression.

[0178] As an example, see Figure 4H , Figure 4H Schematic diagram of the principle of the method for detecting an object in an image provided by the embodiment of the present application. Figure 4HAs shown, the fusion processing of the first target detection result of image 401 (including the first target detection result 21) and the first target detection result of sub-image 402 (including the first target detection result 22) is taken as an example for explanation. Specifically, the fusion processing of the first target detection result of image 401 and the first target detection result of sub-image 402 means that in image 401, the first target detection result 21 in the area 4021 corresponding to the sub-image 402 and the first target detection result 22 of the sub-image 402 are fused. Figure 4H In image 401, the first target detection result 21 in region 4021 corresponding to sub-image 402 includes the position and category of target 211 and the position and category of target 212. The first target detection result 22 of sub-image 402 includes the position and category of target 223, the position and category of target 222, and the position and category of target 221. After the first target detection result of image 401 and the first target detection result of sub-image 402 are fused, the second target detection result 23 corresponding to the image (for region 4021 corresponding to sub-image 402 in image 401) can include the position and category of target 231, the position and category of target 232, and the position and category of target 233. In this way, the positions and categories of three targets can be detected in the fused second target detection result, while in image 401 before fusion, in region 4021 corresponding to sub-image 402, the positions and categories of only two targets can be detected. Therefore, the accuracy of the fused second target detection result is greater than that of the first target detection result of the image before fusion.

[0179] In this way, by performing identification and clustering processing on the density map of the image, multiple clustered target areas in the density map are obtained, and the image is cropped based on the clustered target areas to obtain multiple sub-images that correspond to each other. Since each sub-image includes part of the target in the image, the preliminary detection of the target in the image is achieved through the density map. Then, by performing target detection processing on the sub-image, the target in the sub-image is accurately detected. Since the target in the sub-image is a small-scale target in the image, the detection of the small-scale target in the image is achieved. By performing target detection processing on the image, the large-scale target in the image is detected. By fusing the first target detection result of the sub-image with the first target detection result of the image, the second target detection result obtained can reflect both the large target and the small target in the image, thereby effectively detecting targets of various scales in the image and improving the accuracy of target detection.

[0180] The following is an exemplary application of the embodiment of the present application in an actual application scenario of detecting aerial images.

[0181] In an actual application scenario of detecting aerial images, see Figure 5A , Figure 5A In the aerial images shown, various targets such as vehicles, buildings, trees, street lights, etc. are intricate, and the aerial images collected at different times and locations are different. The scenes are complex and changeable, and the targets in the aerial images (for example, vehicles, buildings, trees, street lights, etc.) are distributed differently, some targets are more concentrated, some targets are more scattered, and the distribution differences between different targets are huge. The target detection method for images provided by the embodiments of the present application can accurately locate targets of different sizes in images under various complex and changeable scenes.

[0182] As an example, see Figure 5B , Figure 5B : is a schematic diagram of the effect of the target detection method for an image provided by an embodiment of the present application. Aerial images can be collected by drone equipment, and target detection is performed on the aerial images by the target detection method for an image provided by an embodiment of the present application to obtain corresponding target images 4, 5, and 6. The position of each target 41 in the aerial image is accurately detected in target image 4, the position of each target 51 in the aerial image is accurately detected in target image 5, and the position of each target 61 in the aerial image is accurately detected in target image 6.

[0183] In some embodiments, Figure 4C Schematic diagram of the principle of the method for detecting an object in an image provided by an embodiment of the present application. Figure 4C , perform density map prediction processing on the image to obtain the density map of the image. Perform identification clustering processing on the density map to obtain multiple clustered target areas in the density map. Perform cropping processing on the image based on the clustered target area to obtain a sub-image. Perform target detection processing on the image and sub-image to obtain the detection result, thereby effectively detecting targets of various scales in the image and improving the accuracy of target detection.

[0184] See also Figure 5C , Figure 5C It is a schematic diagram of the principle of the target detection method for an image provided by an embodiment of the present application. First, a density cropping network is called based on the aerial image to perform density cropping to obtain a sub-image of the aerial image. Specifically, the density cropping network can perform density prediction on the aerial image to obtain a density map corresponding to the aerial image, and determine the clustered target area in the density map, and crop the original image based on the clustered target area in the density map to obtain a sub-image of the aerial image. Then, based on the aerial image and each sub-image of the aerial image, the detection network is called to perform detection processing, and the local detection results corresponding to each sub-image and the global detection results corresponding to the aerial image are obtained respectively. The local detection results and the global detection results are fused to obtain the target detection results corresponding to the aerial image.

[0185] Next, Figure 5C The overall processing flow of the image target detection method provided in the embodiment of the present application is described in detail.

[0186] First of all, Figure 5C The implementation principle of the density clipping network shown in is explained. For details, see Figure 5D Based on the aerial image, the first density map prediction network 51, the second density map prediction network 52 and the third density map prediction network 53 are respectively called to perform density map prediction processing, and the prediction results corresponding to the first density map prediction network 51, the second density map prediction network 52 and the third density map prediction network 53 are merged to obtain a merged result; the merged result is convolved with a convolution kernel size of 1×1 to obtain a density map corresponding to the aerial image. The density map is identified and clustered to obtain a clustered target area, and the aerial image is segmented based on the clustered target area to obtain multiple sub-images corresponding to the aerial image. Among them, the first density map prediction network 51 includes multiple convolution layers with convolution kernel sizes of 9×9 and 7×7, and multiple pooling layers with a pooling kernel size of 2×2; the second density map prediction network 52 includes multiple convolution layers with convolution kernel sizes of 5×5 and 7×7, and multiple pooling layers with a pooling kernel size of 2×2; the third density map prediction network 53 includes multiple convolution layers with convolution kernel sizes of 5×5 and 3×3, and multiple pooling layers with a pooling kernel size of 2×2.

[0187] See also Figure 5D, the Euclidean distance can be used as the loss function to train the first density map prediction network 51, the second density map prediction network 52 and the third density map prediction network 53, that is, based on the sample image, the first density map prediction network 51, the second density map prediction network 52 and the third density map prediction network 53 are respectively called for prediction processing to obtain the sample prediction results of the first density map prediction network 51, the sample prediction results of the second density map prediction network 52 and the sample prediction results of the third density map prediction network 53. The sample prediction results corresponding to the first density map prediction network 51, the sample prediction results corresponding to the second density map prediction network 52 and the sample prediction results corresponding to the third density map prediction network 53 are merged to obtain the sample merged result; the sample merged result is convolved with a convolution kernel size of 1×1 to obtain the predicted density map corresponding to the sample image. The loss function is called based on the predicted density map corresponding to the sample image and the true density map corresponding to the sample image to calculate and obtain the loss value. The first density map prediction network 51, the second density map prediction network 52 and the third density map prediction network 53 are updated based on the loss value to obtain the trained first density map prediction network 51, the trained second density map prediction network 52 and the trained third density map prediction network 53.

[0188] As an example, when the Euclidean distance is used as the loss function (ie, the density loss function described above), the expression of the loss function can be:

[0189]

[0190] Among them, L(θ) represents the loss function, F(X i ,θ) represents the predicted density map, F i represents the true density map, and N represents the number of calls of the density map prediction network.

[0191] See also Figure 5D Based on the aerial image, the trained first density map prediction network 51, the trained second density map prediction network 52 and the trained third density map prediction network 53 are respectively called for prediction processing, and the aerial prediction results of the first density map prediction network 51, the aerial prediction results of the second density map prediction network 52 and the aerial prediction results of the third density map prediction network 53 are respectively obtained, and the aerial prediction results of the first density map prediction network 51, the aerial prediction results of the second density map prediction network 52 and the aerial prediction results of the third density map prediction network 53 are merged to obtain the aerial merged result. The aerial merged result is convolved to obtain the density map corresponding to the aerial image.

[0192] See also Figure 5DAfter obtaining the density map corresponding to the aerial image, the approximate position of the target in the aerial image can be roughly determined through the density map. Since the aerial image is relatively complex and the targets in the aerial image are relatively scattered, the target distribution in the aerial image can be clustered through the Gaussian mixture model to obtain multiple clustered target areas. The aerial image is cropped based on the clustered target areas, and the cropped sub-image of the aerial image is used for subsequent target detection.

[0193] As an example, the probability density function of the multi-dimensional single Gaussian model can be expressed as:

[0194]

[0195] Among them, P(x|θ) represents the probability density function, μ represents the expectation of the multidimensional single Gaussian model, |∑| represents the variance of the multidimensional single Gaussian model, and x represents the density map.

[0196] As an example, a multidimensional Gaussian mixture model is composed of multiple multidimensional single Gaussian models. The probability density function of the multidimensional Gaussian mixture model can be expressed as:

[0197]

[0198] Among them, K represents the number of multidimensional single Gaussian models that make up the multidimensional Gaussian mixture model, k represents the label of the multidimensional single Gaussian model, and α k The weighted coefficients representing the multidimensional single Gaussian model labeled k, Characterize the probability density function of the multidimensional single Gaussian model labeled k.

[0199] Then, yes Figure 5C The implementation principle of the target detection network shown in is explained. For details, see Figure 5E , Figure 5E : This is a schematic diagram of the principle of the target detection network provided by the embodiment of the present application. The target detection network includes a backbone network, a pyramid network and a cascade network, wherein the backbone network includes a first backbone sub-network and a second backbone sub-network, the pyramid network includes a first pyramid network and a second pyramid network, the cascade network includes a pooling network and a region generation network, the first backbone sub-network and the second backbone sub-network include a hierarchical network, the hierarchical network includes a regular hierarchical network and a shifted hierarchical network, the regular hierarchical network includes a regular window, a normalization processing layer and a multi-layer perception layer, and the shifted hierarchical network includes a shifted window, a normalization processing layer and a multi-layer perception layer.

[0200] See also Figure 5E, the backbone network includes a first backbone sub-network (backbone1) and a second backbone sub-network (backbone2), wherein the first backbone sub-network is used to enhance the feature expression capability of the second backbone sub-network. The first backbone sub-network and the second backbone sub-network include a hierarchical network (swin-transformer), wherein the relationship expression between the first backbone sub-network and the second backbone sub-network can be:

[0201]

[0202] Among them, g l-1 Represents the fusion link, the output features of the first backbone sub-network are passed through g l-1 After conversion, it serves as the input of each layer of the second backbone sub-network. Represents the output of the lth level network in the second backbone sub-network, Represents the output of the l-1th level network in the second backbone sub-network.

[0203] See also Figure 5E In order to enhance the feature learning ability of the target detection network for complex scenes, a pyramid network (CBFPN) is used to detect the output of the backbone network, where the pyramid network includes a first pyramid network and a second pyramid network. The detection results of the pyramid network are used for subsequent target positioning and category distinction.

[0204] See also Figure 5E ,The cascade network can effectively improve the target detection performance, and continuously optimize the positioning and classification in multiple stages through multiple pooling networks.

[0205] In this way, by detecting each sub-image through the target detection network, local detection results are obtained, and the specific positions of small targets in the aerial image can be accurately predicted; by detecting the aerial image through the target detection network, local detection results are obtained, and the specific positions of large targets in the aerial image can be accurately predicted. By fusing the local detection results with the global detection results, the fusion process can be achieved by suppressing the maximum values ​​of the local detection results and the global detection results. The target detection results corresponding to the aerial image obtained after fusion can accurately detect the specific positions of small targets in the aerial image, as well as the specific positions of large targets in the aerial image.

[0206] It is understandable that in the embodiments of the present application, the images and other related data involved, when the embodiments of the present application are applied to specific products or technologies, need to obtain user permission or consent, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0207] The following is a description of an exemplary structure of an image object detection device 255 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules in the target detection device 255 of the image stored in the memory 250 may include: a density map prediction module 2551, which is used to perform density map prediction processing on the image to obtain a density map of the image, wherein the density map represents the density distribution of the target in the image; an identification clustering module 2552, which is used to perform identification clustering processing on the density map to obtain multiple clustered target areas in the density map; a cropping module 2553, which is used to perform cropping processing on the image based on the multiple clustered target areas to obtain multiple sub-images corresponding to each other, wherein each sub-image includes multiple targets; a target detection module 2554, which is used to perform target detection processing on each sub-image and the image in turn to obtain a first target detection result corresponding to each sub-image and the image, and a fusion module 2555, which is used to fuse each first target detection result to obtain a second target detection result corresponding to the image.

[0208] In some embodiments, the above-mentioned density map prediction module 2551 is also used to call the trained density map prediction network based on the image to perform density map prediction processing to obtain the density map of the image; wherein the trained density map prediction network is trained in the following manner: based on the image sample, calling the density map prediction network to be trained to perform density map prediction processing to obtain the predicted density map of the image sample; based on the predicted density map of the image sample and the true density map of the image sample, calling the density loss function to obtain the loss value of the image sample; based on the loss value of the image sample, updating the parameters of the density map prediction network to be trained.

[0209] In some embodiments, the above-mentioned density map prediction module 2551 is also used to call the trained density map prediction network to perform the following processing based on the image: perform pooling convolution processing of multiple scales on the image to obtain convolution feature maps of each multiple scales; merge the convolution feature maps of multiple scales to obtain a merged feature map; perform convolution processing on the merged feature map to obtain a density map of the image.

[0210] In some embodiments, the above-mentioned identification clustering module 2552 is also used to identify density points in the density map to obtain the probability distribution of the density points; according to the probability distribution of the density points, the density points that obey the same probability density function are clustered to obtain multiple clustering target areas in the density map.

[0211] In some embodiments, the above-mentioned target detection module 2554 is also used to perform the following processing on any sub-image: perform multi-scale feature extraction processing on the sub-image to obtain multiple features of different scales corresponding to the sub-image; perform a first target detection processing on the features of multiple different scales to obtain a first detection result corresponding to the features of each scale; perform a second target detection processing on each first detection result to obtain a second detection result corresponding to each first detection result, wherein the detection accuracy of the second target detection processing is greater than the detection accuracy of the first target detection processing; and determine the second detection result as the first target detection result.

[0212] In some embodiments, multi-scale feature extraction processing is implemented through a backbone network, which includes a convolutional network, at least one first backbone sub-network with the same structure, and at least one second backbone sub-network; the first backbone sub-network and the second backbone sub-network both include multiple cascaded hierarchical networks, and the sizes of the multiple cascaded hierarchical networks are different; the above-mentioned target detection module 2554 is also used to call the convolutional network for convolution processing based on the sub-image to obtain the convolution processing result of the sub-image; based on the convolution processing result, call multiple cascaded hierarchical networks in the first backbone sub-network for hierarchical processing to obtain the hierarchical processing result of each hierarchical network in the first backbone sub-network; based on the convolution processing result and the hierarchical processing result of each hierarchical network in the first backbone sub-network, call multiple hierarchical networks in the second backbone sub-network for hierarchical processing to obtain the hierarchical processing result of each hierarchical network in the second backbone sub-network; determine the hierarchical processing result of each hierarchical network in the first backbone sub-network and the hierarchical processing result of each hierarchical network in the second backbone sub-network as features of multiple different scales corresponding to the sub-image.

[0213] In some embodiments, the target detection module 2554 is further used to call the first hierarchical network in the first backbone sub-network for hierarchical processing based on the convolution processing result to obtain the first hierarchical processing result of the first backbone sub-network; iterate i to perform the following processing: based on the i-th hierarchical processing result of the first backbone sub-network, call the i+1-th hierarchical network in the first backbone sub-network for hierarchical processing to obtain the i+1-th hierarchical processing result of the first backbone sub-network; wherein, i is an integer variable with increasing values ​​and the value satisfies 1≤i≤N-1, N is the total number of multiple cascaded hierarchical networks, and the size of the i-th hierarchical network is larger than the size of the i+1-th hierarchical network.

[0214] In some embodiments, the target detection module 2554 is further used to sum the convolution processing result, the hierarchical processing result of the i-th hierarchical network of the first backbone subnetwork, and the hierarchical processing result of the 1st hierarchical network of the first backbone subnetwork to obtain the 1st summed processing result; based on the 1st summed processing result, call the 1st hierarchical network in the second backbone subnetwork for hierarchical processing to obtain the 1st hierarchical processing result of the second backbone subnetwork; iterate i and perform the following processing: sum the i-th hierarchical processing result of the first backbone subnetwork and the i-th hierarchical processing result of the second backbone subnetwork to obtain the i+1th summed processing result, call the i+1th hierarchical network in the second backbone subnetwork for hierarchical processing based on the i+1th summed processing result to obtain the i+1th hierarchical processing result of the second backbone subnetwork.

[0215] In some embodiments, the hierarchical network includes a regular hierarchical network and a shifted hierarchical network; the above-mentioned target detection module 2554 is also used to call the i+1th regular hierarchical network in the first backbone subnetwork for hierarchical processing based on the i-th hierarchical processing result of the first backbone subnetwork to obtain the i+1th regular hierarchical processing result of the first backbone subnetwork; based on the i+1th regular hierarchical processing result of the first backbone subnetwork, call the i+1th shifted hierarchical network in the first backbone subnetwork for hierarchical processing, and determine the i+1th shifted hierarchical processing result of the first backbone subnetwork as the i+1th hierarchical processing result of the first backbone subnetwork.

[0216] In some embodiments, the rule hierarchical network includes a rule window, a normalization processing layer and a multi-layer perception layer; the above-mentioned target detection module 2554 is also used to call the normalization processing layer for normalization processing based on the i-th hierarchical processing result of the first backbone sub-network to obtain a first normalization processing result; based on the normalization processing result, call the rule window for self-attention processing to obtain the rule window processing result; sum the i-th hierarchical processing result of the first backbone sub-network and the rule window processing result to obtain the rule window sum processing result; call the normalization processing layer for normalization processing based on the rule window sum processing result to obtain a second normalization processing result; call the multi-layer perception layer for recognition processing based on the second normalization processing result to obtain a first recognition processing result; sum the first recognition processing result and the rule window sum processing result, and determine the obtained first perception layer sum processing result as the i+1th rule hierarchical processing result of the first backbone sub-network.

[0217] In some embodiments, the shifted hierarchical network includes a shifted window, a normalization processing layer and a multi-layer perception layer; the above-mentioned target detection module 2554 is also used to call the normalization processing layer for normalization processing based on the i+1th rule hierarchical processing result of the first backbone sub-network to obtain a third normalization processing result; based on the third normalization processing result, call the shifted window for self-attention processing to obtain a shifted window processing result; sum the i+1th rule hierarchical processing result and the shifted window processing result of the first backbone sub-network to obtain the shifted window sum processing result; call the normalization processing layer for normalization processing based on the shifted window sum processing result to obtain a fourth normalization processing result; call the multi-layer perception layer for recognition processing based on the fourth normalization processing result to obtain a second recognition processing result; sum the second recognition processing result and the shifted window sum processing result, and determine the obtained second perception layer sum processing result as the i+1th shifted hierarchical processing result of the first backbone sub-network.

[0218] In some embodiments, the first target detection processing is implemented by at least one first pyramid network and at least one second pyramid network of the same structure, the number of first pyramid networks is the same as the number of first backbone sub-networks, and the number of second pyramid networks is the same as the number of second backbone sub-networks; the above-mentioned target detection module 2554 is also used to perform the following processing for the features of any scale corresponding to the sub-image: based on the i-th hierarchical processing result of the first backbone sub-network, the first pyramid network is called to perform the first target detection processing, and the first detection result corresponding to the i-th hierarchical processing result of the first backbone sub-network is obtained; based on the i-th hierarchical processing result of the second backbone sub-network, the second pyramid network is called to perform the first target detection processing, and the first detection result corresponding to the i-th hierarchical processing result of the second backbone sub-network is obtained; the first detection result corresponding to the i-th hierarchical processing result of the first backbone sub-network and the first detection result corresponding to the i-th hierarchical processing result of the second backbone sub-network are fused to obtain the first detection result corresponding to the features of each scale.

[0219] In some embodiments, both the first pyramid network and the second pyramid network include multiple convolutional neural networks arranged in order of size; the above-mentioned target detection module 2554 is also used to call the Nth convolutional neural network of the first pyramid network for recognition processing based on the Nth hierarchical processing result of the first backbone subnetwork to obtain the Nth recognition result of the first pyramid network; add the i-th hierarchical processing result of the first backbone subnetwork and the i+1th recognition result of the first pyramid network to obtain the i+1th recognition sum processing result; call the i-th convolutional neural network of the first pyramid network for recognition processing based on the i+1th recognition sum processing result to obtain the i-th recognition result of the first pyramid network, wherein the size of the i-th convolutional neural network is smaller than the size of the i+1th convolutional neural network; determine the i-th recognition result of the first pyramid network as the first detection result corresponding to the i-th hierarchical processing result of the first backbone subnetwork.

[0220] In some embodiments, the second target detection processing is implemented through a cascade network, which includes a region generation network and multiple pooling networks; the above-mentioned target detection module 2554 is also used to perform the following processing for any first detection result: based on the first detection result, calling the region generation network to perform region generation processing to obtain a region generation processing result corresponding to the first detection result; based on the region generation processing result and the first detection result, calling multiple pooling networks to perform pooling processing to obtain a pooling processing result corresponding to each pooling network; and determining the pooling processing result as the second detection result.

[0221] In some embodiments, the target detection module 2554 is further used to call the first pooling network for pooling processing based on the region generation processing result and the first detection result to obtain the first pooling processing result corresponding to the first pooling network; and to call the j+1th pooling network for pooling processing based on the jth pooling processing result and the first detection result to obtain the j+1th pooling processing result corresponding to the j+1th pooling network, wherein 1≤j≤M-1, M is a positive integer greater than or equal to 2, M is the total number of multiple pooling networks, and the accuracy of the j+1th pooling processing result is greater than that of the jth pooling processing result.

[0222] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned method for detecting an object in an image in the embodiment of the present application.

[0223] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the image target detection method provided by the present application embodiment, for example, Figure 3A An object detection method for an image is shown.

[0224] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEP ROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0225] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0226] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0227] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0228] In summary, the embodiments of the present application have the following beneficial effects:

[0229] (1) By clustering the target area obtained by clustering the density map, a sub-image is cropped from the image and identified. Compared with identifying different parts of the original image, the sub-image is identified separately, so that the small-scale targets in the image can be more comprehensively identified. The detection result of the small-scale target (i.e., the first target detection result) is used as a supplement to the overall target detection result of the image (i.e., the second target detection result), thereby taking into account the detection of large and small targets in the image, effectively detecting targets of various scales in the image, and improving the accuracy of target detection.

[0230] (2) By performing density map prediction processing on the image, a density map of the image is obtained, which is convenient for subsequent processing of the density map to crop the image to obtain sub-images, and target detection processing is performed on the sub-images, thereby facilitating subsequent accurate identification of small-scale targets in the image.

[0231] (3) Through the probability distribution of density points in the density map, multiple clustered target areas in the density map are determined, so that the image is preliminarily divided into multiple areas according to the degree of aggregation of the targets in the image, thereby realizing the preliminary detection of the targets in the image through the density map. This facilitates the subsequent cropping of the image to obtain multiple sub-images of the image, providing a good data foundation for subsequent higher-precision target detection.

[0232] (4) The backbone network is used to extract multi-scale features, and features of multiple scales corresponding to the sub-image are obtained. The first target detection is performed through the pyramid network, thereby realizing the first target detection of multiple scales by dividing and conquering features, that is, the first target detection process realizes the first target detection of different scales using features of different scales. The second target detection process is realized through the cascade network, that is, the first target detection results of different scales are refined multiple times using multiple pooling networks, thereby greatly improving the accuracy of target detection.

[0233] (5) By detecting each sub-image through the target detection network, local detection results are obtained, which can accurately predict the specific location of small targets in the aerial image; by detecting the aerial image through the target detection network, local detection results are obtained, which can accurately predict the specific location of large targets in the aerial image. By fusing the local detection results with the global detection results, the fusion process can be achieved by suppressing the maximum value of the local detection results and the global detection results. The target detection results corresponding to the aerial image obtained after fusion can accurately detect the specific location of both the small target and the large target in the aerial image.

[0234] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for detecting an object in an image, characterized in that: The method comprises: Performing density map prediction processing on the image to obtain a density map of the image, wherein the density map represents the density distribution of the target in the image, and the density map prediction processing includes pooling convolution processing of multiple different scales; Performing identification and clustering processing on the density map to obtain a plurality of clustering target areas in the density map; Performing cropping processing on the image based on the multiple clustered target areas to obtain multiple sub-images corresponding to each other, wherein each sub-image includes multiple targets; The following processing is performed for any of the sub-images: Performing multi-scale feature extraction processing on the sub-image to obtain features of multiple different scales corresponding to the sub-image; The multi-scale feature extraction process is implemented by a backbone network, wherein the backbone network includes a convolutional network, at least one first backbone sub-network and at least one second backbone sub-network with the same structure; the first backbone sub-network and the second backbone sub-network both include multiple cascaded hierarchical networks, and the sizes of the multiple cascaded hierarchical networks are different; The following processing is performed for features of any scale corresponding to the sub-image: Based on the i-th hierarchical processing result of the first backbone sub-network, calling the first pyramid network to perform a first target detection process, and obtaining a first detection result corresponding to the i-th hierarchical processing result of the first backbone sub-network; Based on the i-th hierarchical processing result of the second backbone sub-network, calling the second pyramid network to perform a first target detection process, and obtaining a first detection result corresponding to the i-th hierarchical processing result of the second backbone sub-network; Fusing the first detection result corresponding to the i-th hierarchical processing result of the first backbone sub-network and the first detection result corresponding to the i-th hierarchical processing result of the second backbone sub-network to obtain a first detection result corresponding to each of the scale features; The number of the first pyramid networks is the same as the number of the first backbone sub-networks, and the number of the second pyramid networks is the same as the number of the second backbone sub-networks; Performing a second target detection process on each of the first detection results to obtain a second detection result corresponding to each of the first detection results, wherein a detection accuracy of the second target detection process is greater than a detection accuracy of the first target detection process; Determining the second detection result as the first target detection result; Each of the first target detection results is fused to obtain a second target detection result corresponding to the image.

2. The method according to claim 1, characterized in that The performing density map prediction processing on the image to obtain the density map of the image includes: Based on the image, calling the trained density map prediction network to perform density map prediction processing to obtain a density map of the image; The trained density map prediction network is obtained by training in the following way: Based on the image sample, calling the density map prediction network to be trained to perform density map prediction processing to obtain a predicted density map of the image sample; Calling a density loss function based on the predicted density map of the image sample and the true density map of the image sample to obtain a loss value of the image sample; The parameters of the density map prediction network to be trained are updated based on the loss value of the image sample.

3. The method according to claim 2, characterized in that The calling the trained density map prediction network based on the image to perform density map prediction processing to obtain the density map of the image includes: Based on the image, the trained density map prediction network is called to perform the following processing: Performing the pooling convolution processing of the multiple different scales on the image respectively to obtain the convolution feature maps of multiple different scales; Merging the multiple convolution feature maps of different scales to obtain a merged feature map; The combined feature map is convolved to obtain a density map of the image.

4. The method according to claim 1, characterized in that: The performing identification and clustering processing on the density map to obtain a plurality of clustering target areas in the density map includes: Performing identification processing on the density points in the density map to obtain the probability distribution of the density points; According to the probability distribution of the density points, the density points that obey the same probability density function are clustered to obtain a plurality of cluster target areas in the density map.

5. The method according to claim 1, characterized in that: The performing multi-scale feature extraction processing on the sub-image to obtain features of multiple different scales corresponding to the sub-image includes: Calling the convolution network to perform convolution processing based on the sub-image to obtain a convolution processing result of the sub-image; Based on the convolution processing result, calling multiple cascaded hierarchical networks in the first backbone sub-network to perform hierarchical processing, to obtain a hierarchical processing result of each hierarchical network in the first backbone sub-network; Based on the convolution processing result and the hierarchical processing result of each hierarchical network in the first backbone sub-network, multiple hierarchical networks in the second backbone sub-network are called to perform hierarchical processing to obtain the hierarchical processing result of each hierarchical network in the second backbone sub-network; The hierarchical processing results of each hierarchical network in the first backbone sub-network and the hierarchical processing results of each hierarchical network in the second backbone sub-network are determined as features of multiple different scales corresponding to the sub-image.

6. The method according to claim 5, characterized in that The calling of multiple cascaded hierarchical networks in the first backbone sub-network for hierarchical processing based on the convolution processing result to obtain the hierarchical processing result of each hierarchical network in the first backbone sub-network includes: Based on the convolution processing result, calling the first hierarchical network in the first backbone sub-network to perform hierarchical processing to obtain a first hierarchical processing result of the first backbone sub-network; Iteratively perform the following processing: based on the i-th hierarchical processing result of the first backbone subnetwork, call the i+1-th hierarchical network in the first backbone subnetwork for hierarchical processing to obtain the i+1-th hierarchical processing result of the first backbone subnetwork; wherein i is an integer variable with increasing values ​​and the value satisfies 1≤i≤N-1, N is the total number of the multiple cascaded hierarchical networks, and the size of the i-th hierarchical network is larger than the size of the i+1-th hierarchical network.

7. The method according to claim 6, characterized in that The step of calling multiple hierarchical networks in the second backbone subnetwork for hierarchical processing based on the convolution processing result and the hierarchical processing result of each hierarchical network in the first backbone subnetwork to obtain the hierarchical processing result of each hierarchical network in the second backbone subnetwork includes: Adding the convolution processing result and the hierarchical processing result of the i-th level network of the first backbone sub-network to obtain a first summing processing result; Based on the first summing result, calling the first hierarchical network in the second backbone sub-network to perform hierarchical processing, to obtain a first hierarchical processing result of the second backbone sub-network; Iteratively perform the following processing: sum the i-th hierarchical processing result of the first backbone subnetwork and the i-th hierarchical processing result of the second backbone subnetwork to obtain the i+1-th summed processing result, and based on the i+1-th summed processing result, call the i+1-th hierarchical network in the second backbone subnetwork to perform hierarchical processing to obtain the i+1-th hierarchical processing result of the second backbone subnetwork.

8. The method according to claim 7, characterized in that The hierarchical network includes a regular hierarchical network and a shifted hierarchical network; The step of calling the (i+1)th hierarchical network in the first backbone subnetwork to perform hierarchical processing based on the (i)th hierarchical processing result of the first backbone subnetwork to obtain the (i+1)th hierarchical processing result of the first backbone subnetwork includes: Based on the i-th hierarchical processing result of the first backbone sub-network, calling the i+1-th regular hierarchical network in the first backbone sub-network to perform hierarchical processing, to obtain the i+1-th regular hierarchical processing result of the first backbone sub-network; Based on the i+1th rule hierarchical processing result of the first backbone sub-network, the i+1th shifted hierarchical network in the first backbone sub-network is called to perform hierarchical processing, and the obtained i+1th shifted hierarchical processing result of the first backbone sub-network is determined as the i+1th hierarchical processing result of the first backbone sub-network.

9. The method according to claim 8, characterized in that The rule hierarchical network includes a rule window, a normalization processing layer and a multi-layer perception layer; The step of calling the i+1th rule hierarchical network in the first backbone subnetwork to perform hierarchical processing based on the i-th hierarchical processing result of the first backbone subnetwork to obtain the i+1th rule hierarchical processing result of the first backbone subnetwork includes: Calling the normalization processing layer to perform normalization processing based on the i-th hierarchical processing result of the first backbone sub-network to obtain a first normalization processing result; Based on the normalized processing result, the rule window is called to perform self-attention processing to obtain a rule window processing result; Adding the i-th hierarchical processing result of the first backbone sub-network and the rule window processing result to obtain a rule window sum processing result; Calling the normalization processing layer to perform normalization processing based on the rule window sum processing result to obtain a second normalization processing result; Based on the second normalization processing result, calling the multi-layer perception layer to perform recognition processing to obtain a first recognition processing result; The first recognition processing result and the rule window sum processing result are summed up, and the obtained first perception layer sum processing result is determined as the i+1th rule hierarchical processing result of the first backbone sub-network.

10. The method according to claim 8, characterized in that The shifting hierarchical network includes a shifting window, a normalization processing layer and a multi-layer perception layer; The step of calling the i+1th shifted hierarchical network in the first backbone subnetwork for hierarchical processing based on the i+1th rule hierarchical processing result of the first backbone subnetwork, and determining the obtained i+1th shifted hierarchical processing result of the first backbone subnetwork as the i+1th hierarchical processing result of the first backbone subnetwork includes: Calling the normalization processing layer to perform normalization processing based on the i+1th rule hierarchical processing result of the first backbone sub-network to obtain a third normalization processing result; Based on the third normalization processing result, calling the shift window to perform self-attention processing to obtain a shift window processing result; Adding the i+1th rule hierarchical processing result of the first backbone subnetwork and the shift window processing result to obtain a shift window sum processing result; Calling the normalization processing layer to perform normalization processing based on the shift window sum processing result to obtain a fourth normalization processing result; Based on the fourth normalization processing result, calling the multi-layer perception layer to perform recognition processing to obtain a second recognition processing result; The second recognition processing result and the shift window sum processing result are summed up, and the obtained second perception layer sum processing result is determined as the i+1th shift hierarchical processing result of the first backbone subnetwork.

11. The method according to claim 7, characterized in that The second target detection process is implemented by a cascade network, and the cascade network includes a region generation network and multiple pooling networks; The performing a second target detection process on each of the first detection results to obtain a second detection result corresponding to each of the first detection results includes: The following processing is performed for any one of the first detection results: Based on the first detection result, the region generation network is called to perform region generation processing to obtain a region generation processing result corresponding to the first detection result; Based on the region generation processing result and the first detection result, calling the multiple pooling networks to perform pooling processing, and obtaining a pooling processing result corresponding to each of the pooling networks; The pooling processing result is determined as the second detection result.

12. The method according to claim 11, characterized in that The calling the plurality of pooling networks to perform pooling processing based on the region generation processing result and the first detection result to obtain a pooling processing result corresponding to each of the pooling networks includes: Based on the region generation processing result and the first detection result, calling the first pooling network to perform pooling processing, to obtain a first pooling processing result corresponding to the first pooling network; Based on the jth pooling processing result and the first detection result, the j+1th pooling network is called to perform pooling processing to obtain the j+1th pooling processing result corresponding to the j+1th pooling network, wherein 1≤j≤M-1, M is a positive integer greater than or equal to 2, M is the total number of the multiple pooling networks, and the accuracy of the j+1th pooling processing result is greater than that of the jth pooling processing result.

13. An image object detection device, characterized in that: The device comprises: A density map prediction module, used to perform density map prediction processing on an image to obtain a density map of the image, wherein the density map represents the density distribution of the target in the image, and the density map prediction processing includes pooling convolution processing of multiple different scales; An identification and clustering module is used to perform identification and clustering processing on the density map to obtain multiple clustering target areas in the density map; A cropping module, used for cropping the image based on the multiple clustered target areas to obtain multiple sub-images in one-to-one correspondence, wherein each sub-image includes multiple targets; The target detection module is used to perform the following processing on any one of the sub-images: perform multi-scale feature extraction processing on the sub-image to obtain multiple features of different scales corresponding to the sub-image; wherein the multi-scale feature extraction processing is implemented by a backbone network, and the backbone network includes a convolutional network, at least one first backbone sub-network and at least one second backbone sub-network with the same structure; the first backbone sub-network and the second backbone sub-network both include multiple cascaded hierarchical networks, and the sizes of the multiple cascaded hierarchical networks are different; the following processing is performed on the features of any scale corresponding to the sub-image: based on the i-th hierarchical processing result of the first backbone sub-network, call the first pyramid network to perform the first target detection processing, and obtain the first detection result corresponding to the i-th hierarchical processing result of the first backbone sub-network; based on the i-th hierarchical processing result of the second backbone sub-network Call the second pyramid network to perform a first target detection process to obtain a first detection result corresponding to the i-th hierarchical processing result of the second backbone subnetwork; fuse the first detection result corresponding to the i-th hierarchical processing result of the first backbone subnetwork and the first detection result corresponding to the i-th hierarchical processing result of the second backbone subnetwork to obtain a first detection result corresponding to the features of each scale; wherein the number of the first pyramid networks is the same as the number of the first backbone subnetworks, and the number of the second pyramid networks is the same as the number of the second backbone subnetworks; perform a second target detection process on each of the first detection results to obtain a second detection result corresponding to each of the first detection results, wherein the detection accuracy of the second target detection process is greater than the detection accuracy of the first target detection process; determine the second detection result as the first target detection result; A fusion module is used to fuse each of the first target detection results to obtain a second target detection result corresponding to the image.

14. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the method according to any one of claims 1 to 12 when executing the computer executable instructions stored in the memory.

15. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images