Method, device and readable medium for object detection based on millimeter wave images

Abstract

The present invention discloses a method, device and readable medium for object detection based on millimeter wave images. The method comprises the following steps: obtaining millimeter wave images of the front and back of a human body, inputting the millimeter wave images of the front and back of a human body into a trained backbone network to extract front features and back features respectively; interactively fusion-expanding the front features and back features respectively based on an attention mechanism, and inputting the obtained expanded front feature image and / or expanded back feature image into a trained multi-task neural network architecture based on non-homologous data to output full-image object detection results and human skeleton detection results; dividing the image into multiple feature blocks based on the human skeleton detection results, and using an independent detection head to detect each feature block respectively to obtain a secondary detection result, and then performing position mapping on the original image to obtain a block object detection result; performing non-maximum suppression on the block object detection results and the full-image object detection results to obtain a final object detection result.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning, and specifically to a method, device and readable medium for object detection based on millimeter wave images. Background Art

[0002] Millimeter-wave human imaging equipment is harmless to the human body and has strong penetrating power. Its transmission power is less than one thousandth of the electromagnetic radiation of a mobile phone. It can accurately identify items carried by the human body, effectively improve the objectivity, accuracy, and pertinence of inspections, reduce the labor intensity of security inspectors, and improve security inspection efficiency.

[0003] Millimeter waves are shining in today's intelligent security inspection field because of their ability to penetrate clothing without harming the human body. The combination of millimeter wave technology and downstream machine vision algorithms can greatly improve the efficiency and reliability of the security inspection process and reduce the manpower input in the security inspection process. However, the current intelligent security inspection system usually has unsatisfactory detection performance due to problems such as unclear imaging of contraband, insufficient feature mining, and low algorithm performance. It cannot bear the main responsibility of the security inspection process and can only serve as a supplement to manual security inspection.

[0004] The existing millimeter wave contraband detection technology has the following problems:

[0005] 1. In single-angle millimeter wave scanning, the imaging of contraband is usually incomplete and the extracted features are usually not rich enough, resulting in a low detection rate for these contraband.

[0006] 2. Existing detection solutions usually use a model to process one visual task. However, when the application scenario requires multiple tasks to be processed in parallel, the reasoning of multiple models will seriously reduce the algorithm speed, resulting in the algorithm being unable to meet the speed requirements in some real-time scenarios or having very stringent hardware requirements.

[0007] 3. Limited by the development of millimeter wave imaging technology, the imaging of some contraband items is not clear enough (such as powder explosives with low density, lighters with small volume, etc.). These contraband items are often confused with certain background features (such as straps, buttons, muscle lines, imaging noise, etc.), resulting in false detection or missed detection.

[0008] 4. Existing contraband detection methods usually use convolutional neural networks to extract features. This method can obtain the texture information of the target object well, but lacks semantic understanding of the local context of the target. In fact, in the scenario of contraband detection, this local semantic understanding is very necessary, because most contraband will have pockets, tapes / bindings, clothing folds and other features around them. Combining these features for detection can significantly improve the accuracy. Summary of the invention

[0009] In view of the above-mentioned technical problems, the purpose of the embodiments of the present application is to propose a method, device and readable medium for object detection based on millimeter wave images to solve the technical problems mentioned in the above background technology section.

[0010] In a first aspect, an embodiment of the present application provides an object detection method based on millimeter wave images, comprising the following steps:

[0011] S1, obtaining a millimeter wave image of the front side of a human body and a millimeter wave image of the back side of a human body, and inputting the millimeter wave image of the front side of a human body and the millimeter wave image of the back side of a human body into a trained backbone network to extract front features and back features respectively;

[0012] S2, based on the attention mechanism, interactively fuses and expands the front features and the back features to obtain an expanded front feature image and an expanded back feature image;

[0013] S3, inputting the expanded front feature image and / or the expanded back feature image into a trained multi-task neural network architecture based on non-homologous data, and outputting a full-image object detection result and a human skeleton detection result;

[0014] S4, based on the human skeleton detection result, the expanded front feature image and / or the expanded back feature image is divided into multiple feature blocks, and a secondary detection is performed on each feature block using an independent detection head to obtain a secondary detection result, and the secondary detection result is positionally mapped on the expanded front feature image and / or the expanded back feature image to obtain a block object detection result;

[0015] S5, performing non-maximum suppression on the block object detection results and the full-image object detection results to obtain the final object detection results.

[0016] In a specific embodiment, the backbone network adopts a swin-transformer structure.

[0017] In a specific embodiment, step S2 specifically includes:

[0018] The query vector generated by the front feature is compared with the key vector generated by the back feature for similarity, and then the first attention matrix is ​​generated through a softmax operation. The value vector of the back feature of the first attention matrix is ​​adopted, and the sampled back feature is flipped and concatenated with the front feature to obtain an expanded front feature image;

[0019] The query vector generated by the back feature is compared with the key vector generated by the front feature for similarity, and then the second attention matrix is ​​generated through a softmax operation. The value vector of the front feature is sampled by the second attention matrix, and the sampled front feature is flipped and concatenated with the back feature to obtain the expanded back feature image.

[0020] In a specific embodiment, a multi-task neural network architecture based on non-homologous data includes task heads corresponding to object detection tasks, human skeleton detection tasks and other visual tasks, respectively.

[0021] In a specific embodiment, the training process of the multi-task neural network architecture based on non-homologous data in step S3 specifically includes:

[0022] The non-homologous datasets are sampled by a sampler to obtain the task response vector of each sample in the non-homologous dataset. The task response vector corresponds to the annotation of the object detection task, human skeleton detection task and other visual tasks in the dataset. The task response vector of each sample is concatenated to obtain the task response matrix.

[0023] The augmented front feature image and the augmented back feature image corresponding to the non-homologous data set are input into a multi-task neural network architecture based on non-homologous data with multiple task heads, and forward propagation is performed through the multiple task heads respectively to generate the loss value corresponding to each task head, and the loss values ​​corresponding to all the task heads are concatenated into a loss vector;

[0024] The final loss is obtained by multiplying the task response matrix and the loss vector, and backpropagation is performed until the expected effect is achieved or the training end condition is met.

[0025] In a specific embodiment, step S4 specifically includes:

[0026] S41, dividing the expanded front feature image and / or the expanded back feature image into a plurality of regions according to the human skeleton detection result;

[0027] S42, sampling the expanded front feature image and / or the expanded back feature image through a RoIAlign operation based on the coordinates of each region, thereby obtaining a plurality of feature blocks, and recording a starting position of each feature block in the expanded front feature image and / or the expanded back feature image;

[0028] S43, performing secondary detection on each feature block using an independent detection head to obtain a secondary detection result of each feature block;

[0029] S44, adding the detection frame coordinates in the secondary detection result of each feature block to the starting position of each feature block in the expanded front feature image and / or the expanded back feature image to obtain the block object detection result.

[0030] In a specific embodiment, step S43 also includes, in response to the first feature block and the second feature block having symmetrical parts, flipping the second feature block and inputting the second feature block into the detection head corresponding to the first feature block for secondary detection.

[0031] In certain embodiments, the items include contraband.

[0032] In a second aspect, an embodiment of the present application provides an object detection device based on millimeter wave images, comprising:

[0033] A data acquisition module is configured to acquire a millimeter wave image of the front side of a human body and a millimeter wave image of the back side of a human body, and input the millimeter wave image of the front side of a human body and the millimeter wave image of the back side of a human body into a trained backbone network to extract front features and back features respectively;

[0034] The feature expansion module is configured to interactively fuse and expand the front features and the back features based on the attention mechanism to obtain an expanded front feature image and an expanded back feature image;

[0035] A comprehensive detection module is configured to input the expanded front feature image and / or the expanded back feature image into a trained multi-task neural network architecture based on non-homologous data, and output a full-image object detection result and a human skeleton detection result;

[0036] A block detection module is configured to divide the expanded front feature image and / or the expanded back feature image into a plurality of feature blocks based on the human skeleton detection result, and perform secondary detection on each feature block using an independent detection head to obtain a secondary detection result, and perform position mapping on the expanded front feature image and / or the expanded back feature image to obtain a block object detection result;

[0037] The merging module is configured to perform non-maximum suppression on the block object detection results and the full-image object detection results to obtain the final object detection results.

[0038] In a third aspect, an embodiment of the present application provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement a method as described in any implementation manner in the first aspect.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention uses a trained backbone network to extract front features and back features respectively, thereby being able to provide more comprehensive features of the detected object and improve the detection rate of the object.

[0042] (2) The present invention adopts a multi-task neural network architecture based on non-homologous data. Each task in the model has a geographic task head for detection, and the input of each task head can reuse the features extracted by the backbone network, which effectively reduces the training difficulty of the model. Without increasing the labeling cost, the speed of parallel processing of multiple tasks is greatly improved, so that the algorithm meets real-time requirements.

[0043] (3) The present invention also performs human skeleton detection on the subject to be detected, so as to facilitate subsequent segmentation according to the human skeleton detection results, and then the multiple feature blocks obtained after segmentation are respectively subjected to secondary detection by an independent detection head. For each detection head, the input background features are relatively fixed, thereby reducing false detections and improving the detection rate.

[0044] (4) The backbone network of the present invention adopts the swin-transformer structure, which introduces the ability of convolutional networks to focus on local features and form a hierarchical feature pyramid, so that the backbone network combines the advantages of temporal networks and convolutional networks, thereby greatly improving the detection performance.

[0045] (5) The present invention greatly improves the accuracy and efficiency of contraband detection tasks, making it possible to implement a security inspection process led by an intelligent security inspection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 is an exemplary device architecture diagram to which an embodiment of the present application may be applied;

[0048] Figure 2 Schematic diagram of the process of the object detection method based on millimeter wave image according to an embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of the overall process of contraband detection of the object detection method based on millimeter wave images according to an embodiment of the present invention;

[0050] Figure 4Schematic diagram of the process of step S2 of the method for detecting objects based on millimeter wave images according to an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of a training process of a multi-task neural network architecture based on non-homologous data in a method for detecting objects based on millimeter wave images according to an embodiment of the present invention;

[0052] Figure 6 The full-image object detection result and the human skeleton detection result of the object detection method based on millimeter wave images according to an embodiment of the present invention;

[0053] Figure 7 The result of dividing the area according to the human skeleton detection result of the object detection method based on millimeter wave images in an embodiment of the present invention;

[0054] Figure 8 is a schematic diagram of an object detection device based on millimeter wave images according to an embodiment of the present invention;

[0055] Fig. 9 It is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Figure 1 An exemplary device architecture 100 is shown to which the millimeter wave image-based object detection method or millimeter wave image-based object detection device according to the embodiments of the present application can be applied.

[0058] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0059] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various applications, such as data processing applications, file processing applications, etc., can be installed on terminal devices 101, 102, 103.

[0060] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0061] The server 105 may be a server that provides various services, such as a background data processing server that processes files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server may process the acquired files or data and generate processing results.

[0062] It should be noted that the millimeter wave image-based object detection method provided in the embodiment of the present application can be executed by the server 105, or by the terminal devices 101, 102, 103. Accordingly, the millimeter wave image-based object detection device can be set in the server 105, or in the terminal devices 101, 102, 103.

[0063] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided as required. In the case where the processed data does not need to be acquired remotely, the above device architecture may not include a network but only a server or a terminal device.

[0064] Figure 2 An object detection method based on millimeter wave images provided in an embodiment of the present application is shown, comprising the following steps:

[0065] S1, obtaining a millimeter wave image of the front side of a human body and a millimeter wave image of the back side of a human body, and inputting the millimeter wave image of the front side of a human body and the millimeter wave image of the back side of a human body into a trained backbone network to extract front features and back features respectively.

[0066] Specifically, in the embodiments of the present application, millimeter wave images from two angles, the front and back of the human body, are collected to avoid problems such as incomplete imaging of objects scanned by millimeter waves at a single angle, insufficient features extracted, and low detection rate of objects. Figure 3In the embodiments of the present application, the detection of objects in the security inspection scenario is taken as an example of contraband detection, and in other embodiments, it is also applicable to the detection of other objects. In a specific embodiment, the backbone network adopts the swin-transformer structure. In addition to the inherent ability of the transformer to capture global dependencies and strong semantic understanding, the swin-transformer structure also introduces the ability of convolutional networks to focus on local features and form hierarchical feature pyramids through operations such as patching, sliding windows, and shift windows, so that the backbone network combines the advantages of temporal networks and convolutional networks, thereby greatly improving the detection performance. The backbone network can choose any convolutional neural network with multi-level downsampling. The swin-transformer structure is selected here mainly because it is observed that its results are the best, and whether in theory or practice, its ability to capture local dependencies does solve the problems related to local semantic understanding in the scene of contraband detection, because most contraband will appear around pockets, tapes / bindings, clothing wrinkles and other features, and combining these features for detection can significantly improve the accuracy.

[0067] S2, based on the attention mechanism, interactively fuses and expands the front features and the back features respectively to obtain the expanded front feature image and the expanded back feature image.

[0068] In a specific embodiment, step S2 specifically includes:

[0069] The query vector generated by the front feature is compared with the key vector generated by the back feature for similarity, and then the first attention matrix is ​​generated through a softmax operation. The value vector of the back feature of the first attention matrix is ​​adopted, and the sampled back feature is flipped and concatenated with the front feature to obtain an expanded front feature image;

[0070] The query vector generated by the back feature is compared with the key vector generated by the front feature for similarity, and then the second attention matrix is ​​generated through a softmax operation. The value vector of the front feature is sampled by the second attention matrix, and the sampled front feature is flipped and concatenated with the back feature to obtain the expanded back feature image.

[0071] Specifically, refer to Figure 4Taking the front detection as an example, when performing front detection, the back features are also sampled through the attention mechanism. The sampling process is as follows: the query vector (query) generated by the front features is compared with the key vector (key) generated by the back features for similarity, and then the first attention matrix is ​​generated through the softmax operation, and the value vector (value) of the back features is sampled using the attention matrix. Then the sampled back features are horizontally flipped and concatenated with the front features to obtain an expanded front feature image, and finally the expanded front feature image is used for downstream task detection. Similarly, the same principle is used to obtain the expanded back feature image during back detection.

[0072] Therefore, this method first uses a pre-trained backbone network to extract front and back features respectively, and then uses the back features to expand the front features when detecting the front image, and vice versa. This can provide more comprehensive features of the detected object, especially for contraband that only has half of the front and back images when detecting the side of the human body, thereby improving the detection rate of these contraband items.

[0073] S3, inputting the expanded front feature image and / or the expanded back feature image into a trained multi-task neural network architecture based on non-homologous data, and outputting full-image object detection results and human skeleton detection results.

[0074] Specifically, if we follow the traditional neural network training method, we need to ensure that all input training data has labels for all tasks. In the security inspection scenario, taking contraband detection as an example, all data must be labeled for contraband detection, human skeleton detection, and other visual tasks. Obviously, the difficulty of human skeleton detection is much smaller than that of contraband detection, and the amount of data required to train these two tasks is also much smaller than that of contraband detection. Therefore, it is extremely cost-effective to fully label all training data. Therefore, this method adopts a multi-task neural network architecture, that is, each task has its own independent task head, but the input of each task head will reuse the features extracted by the backbone network, that is, share the backbone network.

[0075] In a specific embodiment, the method adopts a multi-task neural network architecture for object detection and human skeleton detection and a hybrid training method for non-homologous data sets. The multi-task neural network architecture based on non-homologous data includes task heads corresponding to object detection tasks, human skeleton detection tasks and other visual tasks. Compared with the traditional one-model-for-one-task architecture, the method adopts a one-model-for-multiple-task architecture, and each task reuses the features extracted from the backbone network, which greatly reduces the number of parameters and inference time of the neural network.

[0076] In order to avoid redundant annotation, this method uses non-homologous datasets during training, that is, data with different types of annotations will be mixed and sampled as training input. This method proposes a method that can train a multi-task neural network architecture using non-homologous datasets. Without increasing the annotation cost, training a multi-task neural network architecture based on non-homologous data greatly improves the speed of parallel processing of multiple tasks and meets real-time requirements. This method additionally marks the task response vector of each sample in the non-homologous dataset during sampling, and uses the task response vector when calculating the loss to ensure that each sample only calculates its loss value in the response task. In a specific embodiment, three visual tasks are assumed: contraband detection, human skeleton detection, and other visual tasks. A task response vector is set for each dataset, the vector length is the number of tasks, for example, 3, and the value in the task response vector consists of 0 or 1. If the dataset has an annotation for the corresponding task, the corresponding position of the task response vector is recorded as 1, otherwise it is 0. For example: if dataset A only annotates the contraband information, its task response vector is [1 0 0]; if dataset B annotates the human skeleton information and other visual task information, its task response vector is [0 1 1].

[0077] In a specific embodiment, the training process of the multi-task neural network architecture based on non-homologous data in step S3 specifically includes:

[0078] The non-homologous datasets are sampled by a sampler to obtain the task response vector of each sample in the non-homologous dataset. The task response vector corresponds to the annotation of the object detection task, human skeleton detection task and other visual tasks in the dataset. The task response vector of each sample is concatenated to obtain the task response matrix.

[0079] The augmented front feature image and the augmented back feature image corresponding to the non-homologous data set are input into a multi-task neural network architecture based on non-homologous data with multiple task heads, and forward propagation is performed through the multiple task heads respectively to generate the loss value corresponding to each task head, and the loss values ​​corresponding to all the task heads are concatenated into a loss vector;

[0080] The final loss is obtained by multiplying the task response matrix and the loss vector, and backpropagation is performed until the expected effect is achieved or the training end condition is met.

[0081] Specifically, refer to Figure 5, the sampler randomly samples from each data set in the non-homologous data set, and records the task response vector of the data set to which each sample belongs. After collecting a certain number of samples (batch size), they are input into the backbone network for feature extraction. The extracted front features or back features are input into the trained multi-task neural network architecture based on non-homologous data after feature expansion in step S2. The three task heads in the multi-task neural network architecture based on non-homologous data are forward propagated respectively. Each task head generates a loss value and forms a loss vector L. The loss vector L is a two-dimensional vector of batch size*3; at the same time, the task response vector of each sample is spliced ​​to obtain the task response matrix M. The task response matrix M is also a two-dimensional task response vector M, and the size is also batch size*3. The final loss is obtained by multiplying the loss vector L and the task response matrix M, and then back propagation is performed. Due to the use of the task response vector, the input of each task head can reuse the features extracted by the same backbone network, effectively reducing the scale of the entire model and reducing the difficulty of model training. The multi-task neural network architecture based on non-homologous data refers to a neural network architecture composed of multiple task heads obtained by the above-mentioned training method, rather than a specific neural network model. Each specific task head can select a suitable existing detection neural network model according to actual detection requirements.

[0082] The extended front feature image and / or extended back feature image extracted from the millimeter wave image of the front of the human body and the millimeter wave image of the back of the human body collected in real time are input into the trained multi-task neural network architecture based on non-homologous data, and the full-image object detection result and the human skeleton detection result are output, such as Figure 6 shown.

[0083] S4, based on the human skeleton detection result, the expanded front feature image and / or the expanded back feature image is divided into multiple feature blocks, and each feature block is subjected to secondary detection using an independent detection head to obtain a secondary detection result, and the secondary detection result is positionally mapped on the expanded front feature image and / or the expanded back feature image to obtain a block object detection result.

[0084] In a specific embodiment, step S4 specifically includes:

[0085] S41, dividing the expanded front feature image and / or the expanded back feature image into a plurality of regions according to the human skeleton detection result;

[0086] S42, sampling the expanded front feature image and / or the expanded back feature image through a RoIAlign operation based on the coordinates of each region, thereby obtaining a plurality of feature blocks, and recording a starting position of each feature block in the expanded front feature image and / or the expanded back feature image;

[0087] S43, performing secondary detection on each feature block using an independent detection head to obtain a secondary detection result of each feature block;

[0088] S44, adding the detection frame coordinates in the secondary detection result of each feature block to the starting position of each feature block in the expanded front feature image and / or the expanded back feature image to obtain the block object detection result.

[0089] In a specific embodiment, step S43 also includes, in response to the first feature block and the second feature block having symmetrical parts, flipping the second feature block and inputting the second feature block into the detection head corresponding to the first feature block for secondary detection.

[0090] Specifically, the method additionally performs human skeleton detection on the detected human body to distinguish the limbs and torso of the detected human body, and then divides the expanded front feature image and / or the expanded back feature image into blocks according to the coordinates of each limb part, which are divided into head and neck, torso, hip / buttocks, upper arm, lower arm, thigh, and calf. Each feature block is then detected by a unique detection head. For each detection head, the input background features are relatively fixed, thereby reducing false detections and improving the detection rate.

[0091] Taking front face detection as an example, the front face is divided into 11 human body regions based on the human skeleton detection results. Figure 7 , including 8 limb regions: 2-4, 4-6, 3-5, 5-7, 8-10, 10-12, 9-11, 11-13; 3 trunk regions: 1-2-3 upward expansion region (head), 2-3-8-9 (trunk), 8-9 (hip / buttocks). Similarly, after flipping the key point coordinates horizontally, the back can also be divided into 11 regions. Using the RoIAlign operation, the features in the expanded front feature image and / or the expanded back feature image are sampled according to the coordinates of the divided regions, thereby obtaining 11 feature blocks of different sizes, and recording the starting position (dx i ,dy i ), which is the coordinate of the upper left corner of the feature block. Use an independent detection head for each feature block for secondary detection. All symmetrical parts share the same detection head, but the input features need to be flipped horizontally. That is, the pairs of parts (2-4, 3-5), (4-6, 5-7), (8-10, 9-11), (10-12, 11-13) share the detection head, and the remaining parts each have a detection head, so there are 7 different detection heads in total. In this way, an independent secondary detection result is obtained for each feature block. The detection box coordinates in the secondary detection result are added to the starting position of each corresponding feature block to obtain the secondary detection result [x min ,y min ,xmax ,y max ]Position mapping in the augmented front feature image and / or the augmented back feature image: [x min +dx i ,y min +dy i ,x max +dx i ,y max +dy i ].

[0092] S5, performing non-maximum suppression on the block object detection results and the full-image object detection results to obtain the final object detection results.

[0093] Specifically, the block object detection results are combined with the full-image object detection results, and the non-maximum suppression algorithm (NMS) is performed to obtain the final object detection results.

[0094] Further references Figure 8 As an implementation of the methods shown in the above figures, the present application provides an embodiment of an object detection device based on millimeter wave images. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0095] The present application embodiment provides an object detection device based on millimeter wave images, including:

[0096] The data acquisition module 1 is configured to acquire a millimeter wave image of the front side of the human body and a millimeter wave image of the back side of the human body, and input the millimeter wave image of the front side of the human body and the millimeter wave image of the back side of the human body into a trained backbone network to extract front features and back features respectively;

[0097] Feature expansion module 2 is configured to interactively fuse and expand the front features and the back features based on the attention mechanism to obtain an expanded front feature image and an expanded back feature image;

[0098] A comprehensive detection module 3 is configured to input the expanded front feature image and / or the expanded back feature image into a trained multi-task neural network architecture based on non-homologous data, and output a full-image object detection result and a human skeleton detection result;

[0099] The block detection module 4 is configured to divide the expanded front feature image and / or the expanded back feature image into a plurality of feature blocks based on the human skeleton detection result, and perform secondary detection on each feature block using an independent detection head to obtain a secondary detection result, and perform position mapping on the expanded front feature image and / or the expanded back feature image to obtain a block object detection result;

[0100] The merging module 5 is configured to perform non-maximum suppression on the block object detection results and the full-image object detection results to obtain the final object detection results.

[0101] Reference below Fig. 9 , which shows an electronic device (eg Figure 1 A schematic diagram of the structure of a computer device 900 (a server or terminal device as shown). Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0102] like Fig. 9 As shown, the computer device 900 includes a central processing unit (CPU) 901 and a graphics processing unit (GPU) 902, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 903 or a program loaded from a storage part 909 to a random access memory (RAM) 904. In the RAM 904, various programs and data required for the operation of the device 900 are also stored. The CPU 901, GPU 902, ROM 903 and RAM 904 are connected to each other through a bus 905. An input / output (I / O) interface 906 is also connected to the bus 905.

[0103] The following components are connected to the I / O interface 906: an input section 907 including a keyboard, a mouse, etc.; an output section 908 including a display such as a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 909 including a hard disk, etc.; and a communication section 910 including a network interface card such as a LAN card, a modem, etc. The communication section 910 performs communication processing via a network such as the Internet. A drive 911 can also be connected to the I / O interface 906 as needed. A removable medium 912, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 911 as needed, so that a computer program read therefrom is installed into the storage section 909 as needed.

[0104] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 910, and / or installed from a removable medium 912. When the computer program is executed by a central processing unit (CPU) 901 and a graphics processing unit (GPU) 902, the above-mentioned functions defined in the method of the present application are executed.

[0105] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, device or component, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution device, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution device, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0106] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] The modules involved in the embodiments described in this application may be implemented by software or hardware, and the modules described may also be set in a processor.

[0109] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a millimeter wave image of the front of the human body and a millimeter wave image of the back of the human body, inputs the millimeter wave image of the front of the human body and the millimeter wave image of the back of the human body into a trained backbone network to extract front features and back features respectively; interactively fuses and expands the front features and back features based on an attention mechanism to obtain an expanded front feature image and an expanded back feature image; inputs the expanded front feature image and / or the expanded back feature image into a trained multi-task neural network architecture based on non-homologous data, and outputs a full-image object detection result and a human skeleton detection result; divides the expanded front feature image and / or the expanded back feature image into multiple feature blocks based on the human skeleton detection result, and uses an independent detection head to perform secondary detection on each feature block to obtain a secondary detection result, and performs position mapping on the secondary detection result on the expanded front feature image and / or the expanded back feature image to obtain a block object detection result; non-maximum suppression is performed on the block object detection result and the full-image object detection result to obtain a final object detection result.

[0110] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for object detection based on millimeter wave images, characterized in that: The following steps are involved: S1, obtaining a millimeter wave image of the front side of a human body and a millimeter wave image of the back side of a human body, and inputting the millimeter wave image of the front side of a human body and the millimeter wave image of the back side of a human body into a trained backbone network to respectively extract front features and back features; S2, interactively fusing and expanding the front features and the back features based on the attention mechanism to obtain an expanded front feature image and an expanded back feature image, specifically including: A query vector generated by the front feature is compared with a key vector generated by the back feature for similarity, and a first attention matrix is ​​generated by a softmax operation, and the value vector of the back feature is sampled by the first attention matrix, and the sampled back feature is flipped and concatenated with the front feature to obtain an expanded front feature image; A query vector generated by the back feature is compared with a key vector generated by the front feature for similarity, and a second attention matrix is ​​generated by a softmax operation, and the second attention matrix is ​​used to sample the value vector of the front feature, and the sampled front feature is flipped and concatenated with the back feature to obtain an expanded back feature image; S3, inputting the expanded front feature image and / or the expanded back feature image into a trained multi-task neural network architecture based on non-homologous data, and outputting a full-image object detection result and a human skeleton detection result; S4, dividing the expanded front feature image and / or the expanded back feature image into a plurality of feature blocks based on the human skeleton detection result, and performing secondary detection on each feature block using an independent detection head to obtain a secondary detection result, and performing position mapping on the expanded front feature image and / or the expanded back feature image to obtain a block object detection result; S5, performing non-maximum suppression on the block object detection results and the full-image object detection results to obtain a final object detection result.

2. The object detection method based on millimeter wave images according to claim 1 is characterized in that: The backbone network adopts a swin-transformer structure.

3. The object detection method based on millimeter wave images according to claim 1 is characterized in that: The multi-task neural network architecture based on non-homologous data includes task heads corresponding to object detection tasks, human skeleton detection tasks and other visual tasks respectively.

4. The object detection method based on millimeter wave images according to claim 3 is characterized in that: The training process of the multi-task neural network architecture based on non-homologous data in step S3 specifically includes: Sampling a non-homologous data set through a sampler, obtaining a task response vector of each sample in the non-homologous data set, wherein the task response vector corresponds to the annotation of the object detection task, the human skeleton detection task and other visual tasks in the data set, and concatenating the task response vector of each sample to obtain a task response matrix; Inputting the expanded front feature image and the expanded back feature image corresponding to the non-homologous data set into a multi-task neural network architecture based on non-homologous data with multiple task heads, performing forward propagation through the multiple task heads respectively, generating a loss value corresponding to each task head, and concatenating the loss values ​​corresponding to all task heads into a loss vector; The task response matrix and the loss vector are dot-multiplied to obtain the final loss, and back-propagation is performed until the expected effect is achieved or the training end condition is met.

5. The object detection method based on millimeter wave images according to claim 1 is characterized in that: The step S4 specifically includes: S41, dividing the expanded front feature image and / or the expanded back feature image into a plurality of regions according to the human skeleton detection result; S42, sampling the expanded front feature image and / or the expanded back feature image through a RoIAlign operation based on the coordinates of each region, thereby obtaining a plurality of feature blocks, and recording a starting position of each feature block in the expanded front feature image and / or the expanded back feature image; S43, performing secondary detection on each feature block using an independent detection head to obtain a secondary detection result of each feature block; S44, adding the detection frame coordinates in the secondary detection result of each feature block to the starting position of each feature block in the expanded front feature image and / or the expanded back feature image to obtain the block object detection result.

6. The object detection method based on millimeter wave images according to claim 5 is characterized in that: The step S43 also includes, in response to the first feature block and the second feature block having symmetrical parts, flipping the second feature block and inputting the second feature block into the detection head corresponding to the first feature block for secondary detection.

7. The object detection method based on millimeter wave images according to any one of claims 1 to 6, characterized in that: The items described include prohibited items.

8. An object detection device based on millimeter wave images, characterized in that: include: A data acquisition module is configured to acquire a millimeter wave image of the front side of a human body and a millimeter wave image of the back side of a human body, and input the millimeter wave image of the front side of a human body and the millimeter wave image of the back side of a human body into a trained backbone network to extract front features and back features respectively; The feature expansion module is configured to interactively fuse and expand the front features and the back features based on the attention mechanism to obtain an expanded front feature image and an expanded back feature image, specifically including: A query vector generated by the front feature is compared with a key vector generated by the back feature for similarity, and a first attention matrix is ​​generated by a softmax operation, and the value vector of the back feature is sampled by the first attention matrix, and the sampled back feature is flipped and concatenated with the front feature to obtain an expanded front feature image; A query vector generated by the back feature is compared with a key vector generated by the front feature for similarity, and a second attention matrix is ​​generated by a softmax operation, and the second attention matrix is ​​used to sample the value vector of the front feature, and the sampled front feature is flipped and concatenated with the back feature to obtain an expanded back feature image; A comprehensive detection module is configured to input the expanded front feature image and / or the expanded back feature image into a trained multi-task neural network architecture based on non-homologous data, and output a full-image object detection result and a human skeleton detection result; A block detection module is configured to divide the expanded front feature image and / or the expanded back feature image into a plurality of feature blocks based on the human skeleton detection result, and perform secondary detection on each feature block using an independent detection head to obtain a secondary detection result, and position map the secondary detection result on the expanded front feature image and / or the expanded back feature image to obtain a block object detection result; The merging module is configured to perform non-maximum suppression on the block object detection results and the full-image object detection results to obtain a final object detection result.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images